xen/vpci: allow unaligned accesses by the hardware domain
PATCH 1/1 xen/vpci: allow unaligned accesses by the hardware domain
It's possible for domains to generate unaligned PCI config space accesses when using ECAM, and hence vPCI should support those at least for the hardware domain. Such unaligned accesses to the PCI config space have been reported to come from ACPI logic.
Relax the checking in vpci_access_allowed() to allow such accesses for the hardware domain, and fix the handling in pci_conf_{read,write}{16,32}() to fulfill them using MMCFG.
MMCFG regions are identity exposed to the hardware domain, and hence such unaligned accesses can only come as a result of the host having MMCFG in the first place, as otherwise MMCFG won't be exposed to the hardware domain either.
--- tools/include/xen-tools/common-macros.h | 2 ++ xen/arch/x86/x86_64/pci.c | 8 ++++---- xen/drivers/vpci/vpci.c | 4 +++- 3 files changed, 9 insertions(+), 5 deletions(-)
tools/include/xen-tools/common-macros.h +2 -0
@@ -68,6 +68,8 @@ })#endif +#define IS_ALIGNED (val, align) (!((val) & ((align) - 1))) +#define ROUNDUP (x, a) (((x) + (a) - 1) & ~((a) - 1))#define ROUNDDOWN (x, a) ((x) & ~((a) - 1))
xen/arch/x86/x86_64/pci.c +4 -4
@@ -26,7 +26,7 @@ uint8_t pci_conf_read8(pci_sbdf_t sbdf, unsigned int reg)uint16_t pci_conf_read16 (pci_sbdf_t sbdf,unsigned int reg) { -if ( sbdf.seg || reg >255 ) +if ( sbdf.seg || reg >255 || !IS_ALIGNED (reg,2 ) ) {uint32_t value;
@@ -39,7 +39,7 @@ uint16_t pci_conf_read16(pci_sbdf_t sbdf, unsigned int reg)uint32_t pci_conf_read32 (pci_sbdf_t sbdf,unsigned int reg) { -if ( sbdf.seg || reg >255 ) +if ( sbdf.seg || reg >255 || !IS_ALIGNED (reg,4 ) ) {uint32_t value;
@@ -60,7 +60,7 @@ void pci_conf_write8(pci_sbdf_t sbdf, unsigned int reg, uint8_t data)void pci_conf_write16 (pci_sbdf_t sbdf,unsigned int reg,uint16_t data) { -if ( sbdf.seg || reg >255 ) +if ( sbdf.seg || reg >255 || !IS_ALIGNED (reg,2 ) )pci_mmcfg_write (sbdf.seg, sbdf.bus, sbdf.devfn, reg,2 , data);else pci_conf_write (PCI_CONF_ADDRESS (sbdf, reg), reg &2 ,2 , data);
@@ -68,7 +68,7 @@ void pci_conf_write16(pci_sbdf_t sbdf, unsigned int reg, uint16_t data)void pci_conf_write32 (pci_sbdf_t sbdf,unsigned int reg,uint32_t data) { -if ( sbdf.seg || reg >255 ) +if ( sbdf.seg || reg >255 || !IS_ALIGNED (reg,4 ) )pci_mmcfg_write (sbdf.seg, sbdf.bus, sbdf.devfn, reg,4 , data);else pci_conf_write (PCI_CONF_ADDRESS (sbdf, reg),0 ,4 , data);
xen/drivers/vpci/vpci.c +3 -1
@@ -685,6 +685,8 @@ void vpci_write(pci_sbdf_t sbdf, unsigned int reg, unsigned int size,/* Helper function to check an access size and alignment on vpci space. */ bool vpci_access_allowed (unsigned int reg,unsigned int len) { +const struct domain *currd = current->domain; +/* Check access size. */ if ( len !=1 && len !=2 && len !=4 && len !=8 )return false;
@@ -696,7 +698,7 @@ bool vpci_access_allowed(unsigned int reg, unsigned int len)#endif /* Check that access is size aligned. */ -if ( (reg & (len -1 )) ) +if ( !is_hardware_domain (currd) && !IS_ALIGNED (reg, len) )return false;return true;
Signature · 1 line
2.53.0
Quoted 4 lines · Roger Pau Monne
It's possible for domains to generate unaligned PCI config space accesses when using ECAM, and hence vPCI should support those at least for the hardware domain. Such unaligned accesses to the PCI config space have been reported to come from ACPI logic.
By this, I presume you mean AML from the DSDT/SSDT ?
Misaligned ECAM accesses have undefined behaviour. On a particular platform this probably means implementation defined, and for an access wholly within a single devices CFG window it might even work. But, misaligning allows you to have one access hitting two devices, and this really can't be a good thing.
I presume "fix your BIOS" isn't on the cards, even if it would be for the greater good?
Quoted 23 lines
Relax the checking in
vpci_access_allowed()to allow such accesses for the hardware domain, and fix the handling in pci_conf_{read,write}{16,32}() to fulfill them using MMCFG.MMCFG regions are identity exposed to the hardware domain, and hence such unaligned accesses can only come as a result of the host having MMCFG in the first place, as otherwise MMCFG won't be exposed to the hardware domain either.
Reported-by Jason Andryuk <[email protected]>Signed-off-by Roger Pau Monné <[email protected]>--- tools/include/xen-tools/common-macros.h | 2 ++ xen/arch/x86/x86_64/pci.c | 8 ++++---- xen/drivers/vpci/vpci.c | 4 +++- 3 files changed, 9 insertions(+), 5 deletions(-)xen/arch/x86/x86_64/pci.c +0
-0
xen/arch/x86/x86_64/pci.c uint8_t pci_conf_read8(pci_sbdf_t sbdf, unsigned int reg) · line 32
Quoted 8 lines
xen/arch/x86/x86_64/pci.c +1
-1@@ -26,7 +26,7 @@ uint8_t pci_conf_read8(pci_sbdf_t sbdf, unsigned int reg)uint16_t pci_conf_read16 (pci_sbdf_t sbdf,unsigned int reg) { -if ( sbdf.seg || reg >255 ) +if ( sbdf.seg || reg >255 || !IS_ALIGNED (reg,2 ) ) {uint32_t value;
Personally, I think this is making a bad situation worse.
Baring quirks (i.e. K10 era), there is never a case where we want to use the IO Ports when we've got ECAM. Furthermore, on AMD systems when we're lacking ECAM we can still access Extended Config Space; something which Xen currently gets wrong in several ways.
The IO ports require a global spinlock in Xen, and then a global resource in hardware just to be able to translate the IO access back into the ECAM access we passed on originally. i.e. from a safety non-interference point of view, you want to veto any use of the IO ports by Xen.
Xen needs to use ECAM, and only fall back to IO Ports if we think there isn't ECAM covering the target sbdf. This will cause (mis)alignment to get fixed automatically. It will also be a substantial perf boost in the general case; all the MSI/MSI-X editing we do (far too frequently) is in Legacy Config Space just uses IO Ports.
~Andrew
Quoted 26 lines · Andrew Cooper
xen/arch/x86/x86_64/pci.c uint8_t pci_conf_read8(pci_sbdf_t sbdf, unsigned int reg) · line 32
Quoted 11 lines · Roger Pau Monne
xen/arch/x86/x86_64/pci.c +1
-1@@ -26,7 +26,7 @@ uint8_t pci_conf_read8(pci_sbdf_t sbdf, unsigned int reg)uint16_t pci_conf_read16 (pci_sbdf_t sbdf,unsigned int reg) { -if ( sbdf.seg || reg >255 ) +if ( sbdf.seg || reg >255 || !IS_ALIGNED (reg,2 ) ) {uint32_t value;Personally, I think this is making a bad situation worse.
Baring quirks (i.e. K10 era), there is never a case where we want to use the IO Ports when we've got ECAM. Furthermore, on AMD systems when we're lacking ECAM we can still access Extended Config Space; something which Xen currently gets wrong in several ways.
The IO ports require a global spinlock in Xen, and then a global resource in hardware just to be able to translate the IO access back into the ECAM access we passed on originally. i.e. from a safety non-interference point of view, you want to veto any use of the IO ports by Xen.
Quoted 5 lines
Xen needs to use ECAM, and only fall back to IO Ports if we think there isn't ECAM covering the target sbdf. This will cause (mis)alignment to get fixed automatically. It will also be a substantial perf boost in the general case; all the MSI/MSI-X editing we do (far too frequently) is in Legacy Config Space just uses IO Ports.
While I agree, that's a bigger change which likely is going to be unsuitable for (immediate) backporting. Nevertheless the cross-device access that the changes as presented could cause needs preventing, by altering the checks at the start of pci_mmcfg_{read,write}().
Jan
Quoted 33 lines · Jan Beulich
Quoted 26 lines · Andrew Cooper
xen/arch/x86/x86_64/pci.c uint8_t pci_conf_read8(pci_sbdf_t sbdf, unsigned int reg) · line 32
Quoted 11 lines · Roger Pau Monne
xen/arch/x86/x86_64/pci.c +1
-1@@ -26,7 +26,7 @@ uint8_t pci_conf_read8(pci_sbdf_t sbdf, unsigned int reg)uint16_t pci_conf_read16 (pci_sbdf_t sbdf,unsigned int reg) { -if ( sbdf.seg || reg >255 ) +if ( sbdf.seg || reg >255 || !IS_ALIGNED (reg,2 ) ) {uint32_t value;Personally, I think this is making a bad situation worse.
Baring quirks (i.e. K10 era), there is never a case where we want to use the IO Ports when we've got ECAM. Furthermore, on AMD systems when we're lacking ECAM we can still access Extended Config Space; something which Xen currently gets wrong in several ways.
The IO ports require a global spinlock in Xen, and then a global resource in hardware just to be able to translate the IO access back into the ECAM access we passed on originally. i.e. from a safety non-interference point of view, you want to veto any use of the IO ports by Xen.
Quoted 5 lines
Xen needs to use ECAM, and only fall back to IO Ports if we think there isn't ECAM covering the target sbdf. This will cause (mis)alignment to get fixed automatically. It will also be a substantial perf boost in the general case; all the MSI/MSI-X editing we do (far too frequently) is in Legacy Config Space just uses IO Ports.
Quoted 4 lines
While I agree, that's a bigger change which likely is going to be unsuitable for (immediate) backporting. Nevertheless the cross-device access that the changes as presented could cause needs preventing, by altering the checks at the start of pci_mmcfg_{read,write}().
Hm, OK. I was planning to be a bit mire strict in vpci_access_allowed(), but I can also adjust pci_mmcfg_{read,write}().
Thanks, Roger.
Quoted 7 lines · Andrew Cooper
Quoted 4 lines · Roger Pau Monne
It's possible for domains to generate unaligned PCI config space accesses when using ECAM, and hence vPCI should support those at least for the hardware domain. Such unaligned accesses to the PCI config space have been reported to come from ACPI logic.
By this, I presume you mean AML from the DSDT/SSDT ?
That's my understanding yes, I can't confirm whether it's from the DSDST/SSDT or maybe one of the OEM tables.
Quoted 5 lines
Misaligned ECAM accesses have undefined behaviour. On a particular platform this probably means implementation defined, and for an access wholly within a single devices CFG window it might even work. But, misaligning allows you to have one access hitting two devices, and this really can't be a good thing.
Well, for the hardware domain I merely wanted to replay the access, but it's true that for access control (that we still do in the hardware domain case) an access targeting two different devices would require extra checking that we don't do.
I think it's possibly best to not relax the check in vpci_access_allowed() as much as I did, and refuse cross-device accesses.
Quoted 2 lines
I presume "fix your BIOS" isn't on the cards, even if it would be for the greater good?
I don't know really, maybe it's possible to get this one fixed. But it's likely Xen would hit this again, and most people would attribute the issue to Xen, because it works with plain Linux, but doesn't when running as a hardware domain (as the unaligned access is ignored or reads all 1s).
We have always said that for the hardware domain we would attempt to get as less as possible in the way of it interacting with devices. I think this is just one more instance of such avoidance of interference from hardware domain accesses.
Quoted 45 lines
Quoted 23 lines
Relax the checking in
vpci_access_allowed()to allow such accesses for the hardware domain, and fix the handling in pci_conf_{read,write}{16,32}() to fulfill them using MMCFG.MMCFG regions are identity exposed to the hardware domain, and hence such unaligned accesses can only come as a result of the host having MMCFG in the first place, as otherwise MMCFG won't be exposed to the hardware domain either.
Reported-by Jason Andryuk <[email protected]>Signed-off-by Roger Pau Monné <[email protected]>--- tools/include/xen-tools/common-macros.h | 2 ++ xen/arch/x86/x86_64/pci.c | 8 ++++---- xen/drivers/vpci/vpci.c | 4 +++- 3 files changed, 9 insertions(+), 5 deletions(-)xen/arch/x86/x86_64/pci.c +0
-0xen/arch/x86/x86_64/pci.c uint8_t pci_conf_read8(pci_sbdf_t sbdf, unsigned int reg) · line 32
Quoted 8 lines
xen/arch/x86/x86_64/pci.c +1
-1@@ -26,7 +26,7 @@ uint8_t pci_conf_read8(pci_sbdf_t sbdf, unsigned int reg)uint16_t pci_conf_read16 (pci_sbdf_t sbdf,unsigned int reg) { -if ( sbdf.seg || reg >255 ) +if ( sbdf.seg || reg >255 || !IS_ALIGNED (reg,2 ) ) {uint32_t value;Personally, I think this is making a bad situation worse.
Baring quirks (i.e. K10 era), there is never a case where we want to use the IO Ports when we've got ECAM. Furthermore, on AMD systems when we're lacking ECAM we can still access Extended Config Space; something which Xen currently gets wrong in several ways.
The IO ports require a global spinlock in Xen, and then a global resource in hardware just to be able to translate the IO access back into the ECAM access we passed on originally. i.e. from a safety non-interference point of view, you want to veto any use of the IO ports by Xen.
Quoted 5 lines
Xen needs to use ECAM, and only fall back to IO Ports if we think there isn't ECAM covering the target sbdf. This will cause (mis)alignment to get fixed automatically. It will also be a substantial perf boost in the general case; all the MSI/MSI-X editing we do (far too frequently) is in Legacy Config Space just uses IO Ports.
I've wondered the same myself for some time, but I was also a bit puzzled by Linux using the same approach, and preferring the IO ports for accesses to the non-extended space.
Overall I think we would widen the hardware domain accesses _before_ ahead of switching PCI accesses to prefer ECAM over IO ports. So that the former can be backported without depending on the latter.
Thanks, Roger.