x86/pci: MMCFG improvements and always use it if available
cover x86/pci: MMCFG improvements and always use it if available
Currently, Xen uses legacy method to access the configuration space unless the access cannot be made with it, where Xen fallbacks to MMCFG. This is not really great, as MMCFG is more flexible and doesn't require a dedicated lock, so it would be preferable to use it whenever possible.
Teddy Astie (2):
x86/pci: Improve pci_mmcfg_{read,write} error handling x86/pci: Prefer using mmcfg for accessing configuration space
xen/arch/x86/x86_64/mmconfig_64.c | 10 +++--- xen/arch/x86/x86_64/pci.c | 52 ++++++++++++++----------------- 2 files changed, 28 insertions(+), 34 deletions(-)
Signature · 10 lines
2.52.0 -- Teddy Astie | Vates XCP-ng Developer XCP-ng & Xen Orchestra - Vates solutions web: https://vates.tech
PATCH 1/2 x86/pci: Improve pci_mmcfg_{read,write} error handling
Return -ENODEV in case no mmcfg information is available instead of -EINVAL, that is also returned when the parameters are incorrect.
It helps us distinguish between incorrect usage and no MMCFG support.
--- xen/arch/x86/x86_64/mmconfig_64.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)
xen/arch/x86/x86_64/mmconfig_64.c +5 -5
@@ -60,15 +60,15 @@ int pci_mmcfg_read(unsigned int seg, unsigned int bus, {char __iomem *addr; + *value =-1 ; +/* Why do we have this when nobody checks it. How about a BUG()!? -AK */ -if (unlikely ((bus >255 ) || (devfn >255 ) || (reg >4095 ))) { -err: *value =-1 ; +if (unlikely ((bus >255 ) || (devfn >255 ) || (reg >4095 )) )return -EINVAL ; - } addr =pci_dev_base (seg, bus, devfn);if (!addr) - goto err; +return -ENODEV ;switch (len) {case 1 :
@@ -96,7 +96,7 @@ int pci_mmcfg_write(unsigned int seg, unsigned int bus, addr =pci_dev_base (seg, bus, devfn);if (!addr) -return -EINVAL ; +return -ENODEV ;switch (len) {case 1 :
Signature · 10 lines
2.52.0 -- Teddy Astie | Vates XCP-ng Developer XCP-ng & Xen Orchestra - Vates solutions web: https://vates.tech
PATCH 2/2 x86/pci: Prefer using mmcfg for accessing configuration space
In reply to Teddy AstieCurrent logic prefer using CFC/CF8 and fallbacks on mmcfg when accessing
Quoted 1 line
255 registers or a non-zero segment. Change the logic to always rely
on mmcfg unless it is not available to avoid locking on pci_config_lock if possible.
---
Are there x86 platforms where MMCFG is the only way to access PCI configuration space ?
xen/arch/x86/x86_64/pci.c | 52 +++++++++++++++++---------------------- 1 file changed, 23 insertions(+), 29 deletions(-)
xen/arch/x86/x86_64/pci.c +23 -29
@@ -14,62 +14,56 @@uint8_t pci_conf_read8 (pci_sbdf_t sbdf,unsigned int reg) {uint32_t value; +int ret =pci_mmcfg_read (sbdf.seg, sbdf.bus, sbdf.devfn, reg,1 , &value); -if ( sbdf.seg || reg >255 ) - { -pci_mmcfg_read (sbdf.seg, sbdf.bus, sbdf.devfn, reg,1 , &value); -return value; - } +if (unlikely (ret == -ENODEV ) && !sbdf.seg && reg <=255 ) +return pci_conf_read (PCI_CONF_ADDRESS (sbdf, reg), reg &3 ,1 ); -return pci_conf_read (PCI_CONF_ADDRESS (sbdf, reg), reg &3 ,1 ); +return value; }uint16_t pci_conf_read16 (pci_sbdf_t sbdf,unsigned int reg) { -if ( sbdf.seg || reg >255 ) - { -uint32_t value; +uint32_t value; +int ret =pci_mmcfg_read (sbdf.seg, sbdf.bus, sbdf.devfn, reg,2 , &value); -pci_mmcfg_read (sbdf.seg, sbdf.bus, sbdf.devfn, reg,2 , &value); -return value; - } +if (unlikely (ret == -ENODEV ) && !sbdf.seg && reg <=255 ) +return pci_conf_read (PCI_CONF_ADDRESS (sbdf, reg), reg &2 ,2 ); -return pci_conf_read (PCI_CONF_ADDRESS (sbdf, reg), reg &2 ,2 ); +return value; }uint32_t pci_conf_read32 (pci_sbdf_t sbdf,unsigned int reg) { -if ( sbdf.seg || reg >255 ) - { -uint32_t value; +uint32_t value; +int ret =pci_mmcfg_read (sbdf.seg, sbdf.bus, sbdf.devfn, reg,4 , &value); -pci_mmcfg_read (sbdf.seg, sbdf.bus, sbdf.devfn, reg,4 , &value); -return value; - } +if (unlikely (ret == -ENODEV ) && !sbdf.seg && reg <=255 ) +return pci_conf_read (PCI_CONF_ADDRESS (sbdf, reg),0 ,4 ); -return pci_conf_read (PCI_CONF_ADDRESS (sbdf, reg),0 ,4 ); +return value; }void pci_conf_write8 (pci_sbdf_t sbdf,unsigned int reg,uint8_t data) { -if ( sbdf.seg || reg >255 ) -pci_mmcfg_write (sbdf.seg, sbdf.bus, sbdf.devfn, reg,1 , data); -else +int ret =pci_mmcfg_write (sbdf.seg, sbdf.bus, sbdf.devfn, reg,1 , data); + +if (unlikely (ret == -ENODEV ) && !sbdf.seg && reg <=255 )pci_conf_write (PCI_CONF_ADDRESS (sbdf, reg), reg &3 ,1 , data); }void pci_conf_write16 (pci_sbdf_t sbdf,unsigned int reg,uint16_t data) { -if ( sbdf.seg || reg >255 ) -pci_mmcfg_write (sbdf.seg, sbdf.bus, sbdf.devfn, reg,2 , data); -else +int ret =pci_mmcfg_write (sbdf.seg, sbdf.bus, sbdf.devfn, reg,2 , data); + +if (unlikely (ret == -ENODEV ) && !sbdf.seg && reg <=255 )pci_conf_write (PCI_CONF_ADDRESS (sbdf, reg), reg &2 ,2 , data); }void pci_conf_write32 (pci_sbdf_t sbdf,unsigned int reg,uint32_t data) { -if ( sbdf.seg || reg >255 ) -pci_mmcfg_write (sbdf.seg, sbdf.bus, sbdf.devfn, reg,4 , data); -else +int ret =pci_mmcfg_write (sbdf.seg, sbdf.bus, sbdf.devfn, reg,4 , data); + +if (unlikely (ret == -ENODEV ) && !sbdf.seg && reg <=255 )pci_conf_write (PCI_CONF_ADDRESS (sbdf, reg),0 ,4 , data); }
Signature · 10 lines
2.52.0 -- Teddy Astie | Vates XCP-ng Developer XCP-ng & Xen Orchestra - Vates solutions web: https://vates.tech
Quoted 8 lines · Teddy Astie
Currently, Xen uses legacy method to access the configuration space unless the access cannot be made with it, where Xen fallbacks to MMCFG. This is not really great, as MMCFG is more flexible and doesn't require a dedicated lock, so it would be preferable to use it whenever possible.
Teddy Astie (2):
x86/pci: Improve pci_mmcfg_{read,write} error handling x86/pci: Prefer using mmcfg for accessing configuration space
AFAICT Linux is using the same approach as Xen to perform PCI accesses. Registers below 256 on segment 0 are accessed using the legacy method (IO ports), while the extended space is accessed using MMCFG. Do you know the reason for this? I fear there might be legacy devices/bridges (or root complexes?) where MMCFG is not working as expected?
I think we need to understand why Xen (and Linux) do it this way so it can be properly justified why it's safe to switch to a different approach.
Thanks, Roger.
Le 07/01/2026 à 18:25, Roger Pau Monné a écrit :
Quoted 10 lines · Teddy Astie
Quoted 8 lines
Currently, Xen uses legacy method to access the configuration space unless the access cannot be made with it, where Xen fallbacks to MMCFG. This is not really great, as MMCFG is more flexible and doesn't require a dedicated lock, so it would be preferable to use it whenever possible.
Teddy Astie (2):
x86/pci: Improve pci_mmcfg_{read,write} error handling x86/pci: Prefer using mmcfg for accessing configuration space
Quoted 7 lines
AFAICT Linux is using the same approach as Xen to perform PCI accesses. Registers below 256 on segment 0 are accessed using the legacy method (IO ports), while the extended space is accessed using MMCFG. Do you know the reason for this? I fear there might be legacy devices/bridges (or root complexes?) where MMCFG is not working as expected?
There is apparently a errata on some K8 chipset according to FreeBSD code that uses MMCFG whenever possible.
https://github.com/freebsd/freebsd-src/blob/main/sys/amd64/pci/pci_cfgreg.c#L261-L277
Quoted 6 lines
I think we need to understand why Xen (and Linux) do it this way so it can be properly justified why it's safe to switch to a different approach.
Thanks, Roger.
--
Teddy Astie | Vates XCP-ng Developer
XCP-ng & Xen Orchestra - Vates solutions
web: https://vates.tech
Quoted 12 lines · Teddy Astie
Le 07/01/2026 à 18:25, Roger Pau Monné a écrit :
Quoted 11 lines · Teddy Astie
Quoted 8 lines
Currently, Xen uses legacy method to access the configuration space unless the access cannot be made with it, where Xen fallbacks to MMCFG. This is not really great, as MMCFG is more flexible and doesn't require a dedicated lock, so it would be preferable to use it whenever possible.
Teddy Astie (2):
x86/pci: Improve pci_mmcfg_{read,write} error handling x86/pci: Prefer using mmcfg for accessing configuration spaceAFAICT Linux is using the same approach as Xen to perform PCI accesses.
I think you mean "Xen inherited it's PCI code from Linux". :)
Quoted 10 lines
Quoted 6 lines
Registers below 256 on segment 0 are accessed using the
legacy method (IO ports), while the extended space is accessed using MMCFG. Do you know the reason for this? I fear there might be legacy devices/bridges (or root complexes?) where MMCFG is not working as expected?There is apparently a errata on some K8 chipset according to FreeBSD code that uses MMCFG whenever possible.
https://github.com/freebsd/freebsd-src/blob/main/sys/amd64/pci/pci_cfgreg.c#L261-L277
Using MMCFG is *far* more efficient than IO ports, not least because we can avoid serialising accesses across the system.
If it really is only some K8's which were the problem then lets go the FreeBSD way. Both Linux and Xen both talk about AMD Fam10h issues, which is the fact that early AMD CPUs only allow MMCFG accesses for MOV-EAX instructions.
There's also loads of code improvements to be had. The current APIs are inefficient to use, and I did some cleanup for them for Trenchboot where there is a 64k limit. (But this can strictly come later).
~Andrew
Quoted 27 lines · Andrew Cooper
Quoted 12 lines · Teddy Astie
Le 07/01/2026 à 18:25, Roger Pau Monné a écrit :
Quoted 11 lines · Teddy Astie
Quoted 8 lines
Currently, Xen uses legacy method to access the configuration space unless the access cannot be made with it, where Xen fallbacks to MMCFG. This is not really great, as MMCFG is more flexible and doesn't require a dedicated lock, so it would be preferable to use it whenever possible.
Teddy Astie (2):
x86/pci: Improve pci_mmcfg_{read,write} error handling x86/pci: Prefer using mmcfg for accessing configuration spaceAFAICT Linux is using the same approach as Xen to perform PCI accesses.
I think you mean "Xen inherited it's PCI code from Linux". :)
Quoted 10 lines
Quoted 6 lines
Registers below 256 on segment 0 are accessed using the
legacy method (IO ports), while the extended space is accessed using MMCFG. Do you know the reason for this? I fear there might be legacy devices/bridges (or root complexes?) where MMCFG is not working as expected?There is apparently a errata on some K8 chipset according to FreeBSD code that uses MMCFG whenever possible.
https://github.com/freebsd/freebsd-src/blob/main/sys/amd64/pci/pci_cfgreg.c#L261-L277
Quoted 7 lines
Using MMCFG is *far* more efficient than IO ports, not least because we can avoid serialising accesses across the system.
If it really is only some K8's which were the problem then lets go the FreeBSD way. Both Linux and Xen both talk about AMD Fam10h issues, which is the fact that early AMD CPUs only allow MMCFG accesses for MOV-EAX instructions.
Sorry if my previous reply made it look the opposite, I'm all fine with switching to MMCFG by default, but we need the K8 workaround, plus this needs to be noted in the commit message.
I wonder if we could use alternative calls to patch MMCFG only access on capable systems, thus removing the check for the fallback legacy access on such systems.
Thanks, Roger.
Re: [PATCH v1 1/2] x86/pci: Improve pci_mmcfg_{read,write} error handling
In reply to Teddy Astiexen/arch/x86/x86_64/mmconfig_64.c int pci_mmcfg_read(unsigned int seg, unsigned int bus, · line 65
Quoted 9 lines · Teddy Astie
xen/arch/x86/x86_64/mmconfig_64.c +2
-0@@ -60,15 +60,15 @@ int pci_mmcfg_read(unsigned int seg, unsigned int bus, {char __iomem *addr; + *value =-1 ; +/* Why do we have this when nobody checks it. How about a BUG()!? -AK */
While it may be okay to retain this comment right here, the next patch the latest should drop it (and its counterpart below). Question though is whether earlier replies to the cover letter won't result in there not actually appearing any users of the return values, in which case the change here would be meaningless. Hence while in principle I'm okay with it, I won't ack it for the time being.
Jan
Re: [PATCH v1 2/2] x86/pci: Prefer using mmcfg for accessing configuration space
In reply to Teddy AstieQuoted 2 lines · Teddy Astie
Current logic prefer using CFC/CF8 and fallbacks on mmcfg when accessing
Quoted 1 line
255 registers or a non-zero segment. Change the logic to always rely
(Minor: Many mail programs, like mine, will mistake a > in the first column as being reply quoting.)
Quoted 6 lines
on mmcfg unless it is not available to avoid locking on
pci_config_lockif possible.Signed-off-by Teddy Astie <[email protected]>---
Are there x86 platforms where MMCFG is the only way to access PCI configuration space ?
If there were, how would that fact be communicated?
xen/arch/x86/x86_64/pci.c line 17
Quoted 7 lines
xen/arch/x86/x86_64/pci.c +1
-0@@ -14,62 +14,56 @@uint8_t pci_conf_read8 (pci_sbdf_t sbdf,unsigned int reg) {uint32_t value; +int ret =pci_mmcfg_read (sbdf.seg, sbdf.bus, sbdf.devfn, reg,1 , &value);
Along the lines of what in particular Roger said in reply to the cover letter, I'm unconvinced we want to slow down (even if just minimally) things by unconditionally making this call (and similar ones below).
Jan