Skip to contentSkip to conversation

xen/vpci: allow unaligned accesses by the hardware domain

PATCH Superseded Roger Pau Monne 3 participants 5 messages Aug 6

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 xen/vpci: allow unaligned accesses by the hardware domain

It's possible for domains to generate unaligned PCI config space accesses when using ECAM, and hence vPCI should support those at least for the hardware domain. Such unaligned accesses to the PCI config space have been reported to come from ACPI logic.

Relax the checking in vpci_access_allowed() to allow such accesses for the hardware domain, and fix the handling in pci_conf_{read,write}{16,32}() to fulfill them using MMCFG.

MMCFG regions are identity exposed to the hardware domain, and hence such unaligned accesses can only come as a result of the host having MMCFG in the first place, as otherwise MMCFG won't be exposed to the hardware domain either.

---
 tools/include/xen-tools/common-macros.h | 2 ++
 xen/arch/x86/x86_64/pci.c               | 8 ++++----
 xen/drivers/vpci/vpci.c                 | 4 +++-
 3 files changed, 9 insertions(+), 5 deletions(-)
tools/include/xen-tools/common-macros.h +2 -0
index 88b4a0e5a693..1f9146b23b0e 100644
@@ -68,6 +68,8 @@
     })
 #endif
 
+#define IS_ALIGNED(val, align) (!((val) & ((align) - 1)))
+
 #define ROUNDUP(x, a) (((x) + (a) - 1) & ~((a) - 1))
 #define ROUNDDOWN(x, a) ((x) & ~((a) - 1))
 
xen/arch/x86/x86_64/pci.c +4 -4
index 8d33429103b9..6298141c3ca7 100644
@@ -26,7 +26,7 @@ uint8_t pci_conf_read8(pci_sbdf_t sbdf, unsigned int reg)
 
 uint16_t pci_conf_read16(pci_sbdf_t sbdf, unsigned int reg)
 {
-    if ( sbdf.seg || reg > 255 )
+    if ( sbdf.seg || reg > 255 || !IS_ALIGNED(reg, 2) )
     {
         uint32_t value;
 
@@ -39,7 +39,7 @@ uint16_t pci_conf_read16(pci_sbdf_t sbdf, unsigned int reg)
 
 uint32_t pci_conf_read32(pci_sbdf_t sbdf, unsigned int reg)
 {
-    if ( sbdf.seg || reg > 255 )
+    if ( sbdf.seg || reg > 255 || !IS_ALIGNED(reg, 4) )
     {
         uint32_t value;
 
@@ -60,7 +60,7 @@ void pci_conf_write8(pci_sbdf_t sbdf, unsigned int reg, uint8_t data)
 
 void pci_conf_write16(pci_sbdf_t sbdf, unsigned int reg, uint16_t data)
 {
-    if ( sbdf.seg || reg > 255 )
+    if ( sbdf.seg || reg > 255 || !IS_ALIGNED(reg, 2) )
         pci_mmcfg_write(sbdf.seg, sbdf.bus, sbdf.devfn, reg, 2, data);
     else
         pci_conf_write(PCI_CONF_ADDRESS(sbdf, reg), reg & 2, 2, data);
@@ -68,7 +68,7 @@ void pci_conf_write16(pci_sbdf_t sbdf, unsigned int reg, uint16_t data)
 
 void pci_conf_write32(pci_sbdf_t sbdf, unsigned int reg, uint32_t data)
 {
-    if ( sbdf.seg || reg > 255 )
+    if ( sbdf.seg || reg > 255 || !IS_ALIGNED(reg, 4) )
         pci_mmcfg_write(sbdf.seg, sbdf.bus, sbdf.devfn, reg, 4, data);
     else
         pci_conf_write(PCI_CONF_ADDRESS(sbdf, reg), 0, 4, data);
xen/drivers/vpci/vpci.c +3 -1
index 0ac9ec8b0475..b4e053bb4946 100644
@@ -685,6 +685,8 @@ void vpci_write(pci_sbdf_t sbdf, unsigned int reg, unsigned int size,
 /* Helper function to check an access size and alignment on vpci space. */
 bool vpci_access_allowed(unsigned int reg, unsigned int len)
 {
+    const struct domain *currd = current->domain;
+
     /* Check access size. */
     if ( len != 1 && len != 2 && len != 4 && len != 8 )
         return false;
@@ -696,7 +698,7 @@ bool vpci_access_allowed(unsigned int reg, unsigned int len)
 #endif
 
     /* Check that access is size aligned. */
-    if ( (reg & (len - 1)) )
+    if ( !is_hardware_domain(currd) && !IS_ALIGNED(reg, len) )
         return false;
 
     return true;
Signature · 1 line
2.53.0
#1 lore raw
Quoted 4 lines · Roger Pau Monne

It's possible for domains to generate unaligned PCI config space accesses when using ECAM, and hence vPCI should support those at least for the hardware domain. Such unaligned accesses to the PCI config space have been reported to come from ACPI logic.

By this, I presume you mean AML from the DSDT/SSDT ?

Misaligned ECAM accesses have undefined behaviour.  On a particular platform this probably means implementation defined, and for an access wholly within a single devices CFG window it might even work.  But, misaligning allows you to have one access hitting two devices, and this really can't be a good thing.

I presume "fix your BIOS" isn't on the cards, even if it would be for the greater good?

Quoted 23 lines

Relax the checking in vpci_access_allowed() to allow such accesses for the hardware domain, and fix the handling in pci_conf_{read,write}{16,32}() to fulfill them using MMCFG.

MMCFG regions are identity exposed to the hardware domain, and hence such unaligned accesses can only come as a result of the host having MMCFG in the first place, as otherwise MMCFG won't be exposed to the hardware domain either.

---
 tools/include/xen-tools/common-macros.h | 2 ++
 xen/arch/x86/x86_64/pci.c               | 8 ++++----
 xen/drivers/vpci/vpci.c                 | 4 +++-
 3 files changed, 9 insertions(+), 5 deletions(-)
xen/arch/x86/x86_64/pci.c +0 -0
index 8d33429103b9..6298141c3ca7 100644

xen/arch/x86/x86_64/pci.c uint8_t pci_conf_read8(pci_sbdf_t sbdf, unsigned int reg) · line 32

Quoted 8 lines
xen/arch/x86/x86_64/pci.c +1 -1
@@ -26,7 +26,7 @@ uint8_t pci_conf_read8(pci_sbdf_t sbdf, unsigned int reg)
 
 uint16_t pci_conf_read16(pci_sbdf_t sbdf, unsigned int reg)
 {
-    if ( sbdf.seg || reg > 255 )
+    if ( sbdf.seg || reg > 255 || !IS_ALIGNED(reg, 2) )
     {
         uint32_t value;
 

Personally, I think this is making a bad situation worse.

Baring quirks (i.e. K10 era), there is never a case where we want to use the IO Ports when we've got ECAM.  Furthermore, on AMD systems when we're lacking ECAM we can still access Extended Config Space; something which Xen currently gets wrong in several ways.

The IO ports require a global spinlock in Xen, and then a global resource in hardware just to be able to translate the IO access back into the ECAM access we passed on originally.  i.e. from a safety non-interference point of view, you want to veto any use of the IO ports by Xen.

Xen needs to use ECAM, and only fall back to IO Ports if we think there isn't ECAM covering the target sbdf.  This will cause (mis)alignment to get fixed automatically.  It will also be a substantial perf boost in the general case; all the MSI/MSI-X editing we do (far too frequently) is in Legacy Config Space just uses IO Ports.

~Andrew

#2 lore raw
Quoted 26 lines · Andrew Cooper

xen/arch/x86/x86_64/pci.c uint8_t pci_conf_read8(pci_sbdf_t sbdf, unsigned int reg) · line 32

Quoted 11 lines · Roger Pau Monne
xen/arch/x86/x86_64/pci.c +1 -1
@@ -26,7 +26,7 @@ uint8_t pci_conf_read8(pci_sbdf_t sbdf, unsigned int reg)
 
 uint16_t pci_conf_read16(pci_sbdf_t sbdf, unsigned int reg)
 {
-    if ( sbdf.seg || reg > 255 )
+    if ( sbdf.seg || reg > 255 || !IS_ALIGNED(reg, 2) )
     {
         uint32_t value;
 

Personally, I think this is making a bad situation worse.

Baring quirks (i.e. K10 era), there is never a case where we want to use the IO Ports when we've got ECAM.  Furthermore, on AMD systems when we're lacking ECAM we can still access Extended Config Space; something which Xen currently gets wrong in several ways.

The IO ports require a global spinlock in Xen, and then a global resource in hardware just to be able to translate the IO access back into the ECAM access we passed on originally.  i.e. from a safety non-interference point of view, you want to veto any use of the IO ports by Xen.

Quoted 5 lines

Xen needs to use ECAM, and only fall back to IO Ports if we think there isn't ECAM covering the target sbdf.  This will cause (mis)alignment to get fixed automatically.  It will also be a substantial perf boost in the general case; all the MSI/MSI-X editing we do (far too frequently) is in Legacy Config Space just uses IO Ports.

While I agree, that's a bigger change which likely is going to be unsuitable for (immediate) backporting. Nevertheless the cross-device access that the changes as presented could cause needs preventing, by altering the checks at the start of pci_mmcfg_{read,write}().

Jan

#3 lore raw
Quoted 33 lines · Jan Beulich
Quoted 26 lines · Andrew Cooper

xen/arch/x86/x86_64/pci.c uint8_t pci_conf_read8(pci_sbdf_t sbdf, unsigned int reg) · line 32

Quoted 11 lines · Roger Pau Monne
xen/arch/x86/x86_64/pci.c +1 -1
@@ -26,7 +26,7 @@ uint8_t pci_conf_read8(pci_sbdf_t sbdf, unsigned int reg)
 
 uint16_t pci_conf_read16(pci_sbdf_t sbdf, unsigned int reg)
 {
-    if ( sbdf.seg || reg > 255 )
+    if ( sbdf.seg || reg > 255 || !IS_ALIGNED(reg, 2) )
     {
         uint32_t value;
 

Personally, I think this is making a bad situation worse.

Baring quirks (i.e. K10 era), there is never a case where we want to use the IO Ports when we've got ECAM.  Furthermore, on AMD systems when we're lacking ECAM we can still access Extended Config Space; something which Xen currently gets wrong in several ways.

The IO ports require a global spinlock in Xen, and then a global resource in hardware just to be able to translate the IO access back into the ECAM access we passed on originally.  i.e. from a safety non-interference point of view, you want to veto any use of the IO ports by Xen.

Quoted 5 lines

Xen needs to use ECAM, and only fall back to IO Ports if we think there isn't ECAM covering the target sbdf.  This will cause (mis)alignment to get fixed automatically.  It will also be a substantial perf boost in the general case; all the MSI/MSI-X editing we do (far too frequently) is in Legacy Config Space just uses IO Ports.

Quoted 4 lines

While I agree, that's a bigger change which likely is going to be unsuitable for (immediate) backporting. Nevertheless the cross-device access that the changes as presented could cause needs preventing, by altering the checks at the start of pci_mmcfg_{read,write}().

Hm, OK. I was planning to be a bit mire strict in vpci_access_allowed(), but I can also adjust pci_mmcfg_{read,write}().

Thanks, Roger.

#4 lore raw
In reply to Andrew Cooper
Quoted 7 lines · Andrew Cooper
Quoted 4 lines · Roger Pau Monne

It's possible for domains to generate unaligned PCI config space accesses when using ECAM, and hence vPCI should support those at least for the hardware domain. Such unaligned accesses to the PCI config space have been reported to come from ACPI logic.

By this, I presume you mean AML from the DSDT/SSDT ?

That's my understanding yes, I can't confirm whether it's from the DSDST/SSDT or maybe one of the OEM tables.

Quoted 5 lines

Misaligned ECAM accesses have undefined behaviour.  On a particular platform this probably means implementation defined, and for an access wholly within a single devices CFG window it might even work.  But, misaligning allows you to have one access hitting two devices, and this really can't be a good thing.

Well, for the hardware domain I merely wanted to replay the access, but it's true that for access control (that we still do in the hardware domain case) an access targeting two different devices would require extra checking that we don't do.

I think it's possibly best to not relax the check in vpci_access_allowed() as much as I did, and refuse cross-device accesses.

Quoted 2 lines

I presume "fix your BIOS" isn't on the cards, even if it would be for the greater good?

I don't know really, maybe it's possible to get this one fixed. But it's likely Xen would hit this again, and most people would attribute the issue to Xen, because it works with plain Linux, but doesn't when running as a hardware domain (as the unaligned access is ignored or reads all 1s).

We have always said that for the hardware domain we would attempt to get as less as possible in the way of it interacting with devices. I think this is just one more instance of such avoidance of interference from hardware domain accesses.

Quoted 45 lines
Quoted 23 lines

Relax the checking in vpci_access_allowed() to allow such accesses for the hardware domain, and fix the handling in pci_conf_{read,write}{16,32}() to fulfill them using MMCFG.

MMCFG regions are identity exposed to the hardware domain, and hence such unaligned accesses can only come as a result of the host having MMCFG in the first place, as otherwise MMCFG won't be exposed to the hardware domain either.

---
 tools/include/xen-tools/common-macros.h | 2 ++
 xen/arch/x86/x86_64/pci.c               | 8 ++++----
 xen/drivers/vpci/vpci.c                 | 4 +++-
 3 files changed, 9 insertions(+), 5 deletions(-)
xen/arch/x86/x86_64/pci.c +0 -0
index 8d33429103b9..6298141c3ca7 100644

xen/arch/x86/x86_64/pci.c uint8_t pci_conf_read8(pci_sbdf_t sbdf, unsigned int reg) · line 32

Quoted 8 lines
xen/arch/x86/x86_64/pci.c +1 -1
@@ -26,7 +26,7 @@ uint8_t pci_conf_read8(pci_sbdf_t sbdf, unsigned int reg)
 
 uint16_t pci_conf_read16(pci_sbdf_t sbdf, unsigned int reg)
 {
-    if ( sbdf.seg || reg > 255 )
+    if ( sbdf.seg || reg > 255 || !IS_ALIGNED(reg, 2) )
     {
         uint32_t value;
 

Personally, I think this is making a bad situation worse.

Baring quirks (i.e. K10 era), there is never a case where we want to use the IO Ports when we've got ECAM.  Furthermore, on AMD systems when we're lacking ECAM we can still access Extended Config Space; something which Xen currently gets wrong in several ways.

The IO ports require a global spinlock in Xen, and then a global resource in hardware just to be able to translate the IO access back into the ECAM access we passed on originally.  i.e. from a safety non-interference point of view, you want to veto any use of the IO ports by Xen.

Quoted 5 lines

Xen needs to use ECAM, and only fall back to IO Ports if we think there isn't ECAM covering the target sbdf.  This will cause (mis)alignment to get fixed automatically.  It will also be a substantial perf boost in the general case; all the MSI/MSI-X editing we do (far too frequently) is in Legacy Config Space just uses IO Ports.

I've wondered the same myself for some time, but I was also a bit puzzled by Linux using the same approach, and preferring the IO ports for accesses to the non-extended space.

Overall I think we would widen the hardware domain accesses _before_ ahead of switching PCI accesses to prefer ECAM over IO ports. So that the former can be backported without depending on the latter.

Thanks, Roger.