PCI: vmd: Add feature to scan BIOS enumerated devices.
Subsystems (3)
Nirmal Patel
maintainerPATCH v3 1/1 PCI: vmd: Add feature to scan BIOS enumerated devices.
Newer VMD with device ID 0x28c1 has unique settings compared to its predecessor where BIOS enumerates the entire VMD device tree and assigns respective configurations.
VMD configuration BAR0 carries over from GNR legacy VMD as the mechanism to access the configuration space of the devices owned by VMD. The size of this window is fixed at 256 MB, where each function consumes 4 KB and every bus consumes 1 MB.
The shadow and scratchpad registers have been relocated from the VMD configuration space to the VMD MMIO space in VMD BAR4/BAR5, otherwise refers to as MEMBAR2 or MSI-X bar.
VMD MSI-X remapping enable/disable is no longer supported.
All the VMD driver code needs to do is to obtain bus hide range along with shadow register values set by BIOS and perform a bus scan.
The patch also involves small refactoring of vmd_enable_domain function.
---
v3 : Hard code configbar .end to 0xff same as probe.c; Adjust membar2
offset to accommodate more registers in 28C1. Remove redundant IORESOURCE_MEM check in vmd_prepare_offsets_and_bus.
v2 : Using PCI features flag instead of devic ID and fixing corner cases
for vmd_remove_irq.
--- drivers/pci/controller/vmd.c | 172 +++++++++++++++++++++++++++-------- include/linux/pci_ids.h | 1 + 2 files changed, 134 insertions(+), 39 deletions(-)
drivers/pci/controller/vmd.c +133 -39
@@ -37,6 +37,12 @@#define MB2_SHADOW_OFFSET 0x2000#define MB2_SHADOW_SIZE 16 +/* DMR BAR4 register offsets */ +#define SHADOW_MEMBAR1_28C1 0x2818/* MEMBAR1 physical address */ +#define SHADOW_MEMBAR2_28C1 0x2820/* MEMBAR2 physical address */ +#define BASE_ID_REG_28C1 0x2840 +#define MEMBAR2_OFFSET_28C1 0x30d0 +enum vmd_features {/* * Device may contain registers which hint the physical location of the
@@ -77,6 +83,15 @@ enum vmd_features {* proper power management of the SoC. */ VMD_FEAT_BIOS_PM_QUIRK = (1 <<5 ), + +/* +* Newer VMD with device ID 0x28c1 has unique settings compared to its +* predecessor where BIOS enumerates the entire VMD device tree and +* stores respective configurations including bus start range and +* shadow registers in VMD MMIO space in VMD BAR4/BAR5, otherwise refers +* to as MEMBAR2 or MSI-X bar. +*/ + VMD_FEAT_USE_BIOS_INFO = (1 << 6), };#define VMD_BIOS_PM_QUIRK_LTR 0x1003/* 3145728 ns */
@@ -142,6 +157,7 @@ struct vmd_dev {u8 first_vec;char *name;int instance; +unsigned long features; };static inline struct vmd_dev *vmd_from_bus (struct pci_bus *bus)
@@ -374,6 +390,8 @@ static void vmd_set_msi_remapping(struct vmd_dev *vmd, bool enable)static void vmd_remove_irq_domain (struct vmd_dev *vmd) { +if (!!(vmd->features &VMD_FEAT_USE_BIOS_INFO )) +return ;/* * Some production BIOS won't enable remapping between soft reboots. * Ensure remapping is restored before unloading the driver.
@@ -393,7 +411,12 @@ static void __iomem *vmd_cfg_addr(struct vmd_dev *vmd, struct pci_bus *bus, unsigned int devfn, int reg, int len) { unsigned int busnr_ecam = bus->number - vmd->busn_start; -u32 offset =PCIE_ECAM_OFFSET (busnr_ecam, devfn, reg); +u32 offset; + +if (!!(vmd->features &VMD_FEAT_USE_BIOS_INFO )) + busnr_ecam = bus->number; + + offset =PCIE_ECAM_OFFSET (busnr_ecam, devfn, reg);if (offset + len >=resource_size (&vmd->dev->resource[VMD_CFGBAR ]))return NULL ;
@@ -661,6 +684,46 @@ static int vmd_get_bus_number_start(struct vmd_dev *vmd)return 0 ; } +static int vmd_get_bus_info_from_bar4 (struct vmd_dev *vmd, +resource_size_t *offset1, +resource_size_t *offset2) +{ +u64 phys1, phys2, bar4_2840; +void __iomem *bar4; +u32 base_id; +u8 base_bus; + + + bar4 =pci_ioremap_bar (vmd->dev,4 ); +if (!bar4) +return -ENOMEM ; + +/* Read shadow registers for MEMBAR1 and MEMBAR2 physical addresses. */ + phys1 =readq (bar4 +SHADOW_MEMBAR1_28C1 ); + phys2 =readq (bar4 +SHADOW_MEMBAR2_28C1 ); + +/* +* Read and set bus start number from Base ID register. +* 24-bit Base ID register is part of 64-bit shadowed reqid hide +* range register and holds segment, bus, device and function. +*/ + bar4_2840 =readq (bar4 +BASE_ID_REG_28C1 ); + base_id = bar4_2840 &0xFFFFFF ; + base_bus = base_id >>8 ; + vmd->busn_start = base_bus; + +/* Calculate offsets like vmd_get_phys_offsets() does. */ +if (phys1) + *offset1 = vmd->dev->resource[VMD_MEMBAR1 ].start - + (phys1 &PCI_BASE_ADDRESS_MEM_MASK ); +if (phys2) + *offset2 = vmd->dev->resource[VMD_MEMBAR2 ].start - + (phys2 &PCI_BASE_ADDRESS_MEM_MASK ); + +pci_iounmap (vmd->dev, bar4); +return 0 ; +} +static irqreturn_t vmd_irq (int irq,void *data) {struct vmd_irq_list *irqs = data;
@@ -711,6 +774,53 @@ static int vmd_alloc_irqs(struct vmd_dev *vmd)return 0 ; } +static int vmd_prepare_offsets_and_bus (struct vmd_dev *vmd, +unsigned long features, +resource_size_t *membar2_offset, +resource_size_t *offset1, +resource_size_t *offset2) +{ +int ret; + +/* +* Shadow registers may exist in certain VMD device ids which allow +* guests to correctly assign host physical addresses to the root ports +* and child devices. These registers will either return the host value +* or 0, depending on an enable bit in the VMD device. +*/ +/* +* For certain VMD devices (i.e. 0x28C1), BIOS places device info +* in BAR4 shadow registers to determine the base bus number and memory +* offsets. +*/ +if (features &VMD_FEAT_USE_BIOS_INFO ) { + *membar2_offset =MEMBAR2_OFFSET_28C1 ; + ret =vmd_get_bus_info_from_bar4 (vmd, offset1, offset2); +if (ret) +return ret; + }else if (features &VMD_FEAT_HAS_MEMBAR_SHADOW ) { + *membar2_offset =MB2_SHADOW_OFFSET +MB2_SHADOW_SIZE ; + ret =vmd_get_phys_offsets (vmd, true, offset1, offset2); +if (ret) +return ret; + }else if (features &VMD_FEAT_HAS_MEMBAR_SHADOW_VSCAP ) { + ret =vmd_get_phys_offsets (vmd, false, offset1, offset2); +if (ret) +return ret; + } + +/* +* Certain VMD devices may have a root port configuration option which +* limits the bus range to between 0-127, 128-255, or 224-255. +*/ +if (features &VMD_FEAT_HAS_BUS_RESTRICTIONS ) { + ret =vmd_get_bus_number_start (vmd); +if (ret) +return ret; + } +return 0 ; +} +/* * Since VMD is an aperture to regular PCIe root ports, only allow it to * control features that the OS is allowed to control on the physical PCI bus.
@@ -784,38 +894,16 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)struct pci_dev *dev;int ret; -/* -* Shadow registers may exist in certain VMD device ids which allow -* guests to correctly assign host physical addresses to the root ports -* and child devices. These registers will either return the host value -* or 0, depending on an enable bit in the VMD device. -*/ -if (features &VMD_FEAT_HAS_MEMBAR_SHADOW ) { - membar2_offset =MB2_SHADOW_OFFSET +MB2_SHADOW_SIZE ; - ret =vmd_get_phys_offsets (vmd, true, &offset[0 ], &offset[1 ]); -if (ret) -return ret; - }else if (features &VMD_FEAT_HAS_MEMBAR_SHADOW_VSCAP ) { - ret =vmd_get_phys_offsets (vmd, false, &offset[0 ], &offset[1 ]); -if (ret) -return ret; - } - -/* -* Certain VMD devices may have a root port configuration option which -* limits the bus range to between 0-127, 128-255, or 224-255 -*/ -if (features &VMD_FEAT_HAS_BUS_RESTRICTIONS ) { - ret =vmd_get_bus_number_start (vmd); -if (ret) -return ret; - } + ret =vmd_prepare_offsets_and_bus (vmd, features, &membar2_offset, + &offset[0 ], &offset[1 ]); +if (ret) +return ret; res = &vmd->dev->resource[VMD_CFGBAR ]; vmd->resources[0 ] = (struct resource ) { .name ="VMD CFGBAR" , .start = vmd->busn_start, - .end = vmd->busn_start + (resource_size (res) >>20 ) -1 , + .end =0xff , .flags =IORESOURCE_BUS |IORESOURCE_PCI_FIXED , };
@@ -868,19 +956,21 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)* acceptable because the guest is usually CPU-limited and MSI * remapping doesn't become a performance bottleneck. */ -if (!(features &VMD_FEAT_CAN_BYPASS_MSI_REMAP ) || - offset[0 ] || offset[1 ]) { - ret =vmd_alloc_irqs (vmd); -if (ret) -return ret; +if (!(features &VMD_FEAT_USE_BIOS_INFO )) { +if (!(features &VMD_FEAT_CAN_BYPASS_MSI_REMAP ) || + offset[0 ] || offset[1 ]) { + ret =vmd_alloc_irqs (vmd); +if (ret) +return ret; -vmd_set_msi_remapping (vmd, true); +vmd_set_msi_remapping (vmd, true); - ret =vmd_create_irq_domain (vmd); -if (ret) -return ret; - }else { -vmd_set_msi_remapping (vmd, false); + ret =vmd_create_irq_domain (vmd); +if (ret) +return ret; + }else { +vmd_set_msi_remapping (vmd, false); + } }pci_add_resource (&resources, &vmd->resources[0 ]);
@@ -998,6 +1088,7 @@ static int vmd_probe(struct pci_dev *dev, const struct pci_device_id *id) vmd->dev = dev; vmd->sysdata.domain =PCI_DOMAIN_NR_NOT_SET ; + vmd->features = features; vmd->instance =ida_alloc (&vmd_instance_ida,GFP_KERNEL );if (vmd->instance <0 )return vmd->instance;
@@ -1114,6 +1205,9 @@ static const struct pci_device_id vmd_ids[] = { .driver_data = VMD_FEAT_HAS_MEMBAR_SHADOW | VMD_FEAT_HAS_BUS_RESTRICTIONS | VMD_FEAT_CAN_BYPASS_MSI_REMAP,}, + {PCI_VDEVICE(INTEL, PCI_DEVICE_ID_INTEL_VMD_28C1), + .driver_data =VMD_FEAT_HAS_MEMBAR_SHADOW | + VMD_FEAT_USE_BIOS_INFO,}, {PCI_VDEVICE(INTEL, 0x467f), .driver_data = VMD_FEATS_CLIENT,}, {PCI_VDEVICE(INTEL, 0x4c3d),
include/linux/pci_ids.h +1 -0
@@ -2889,6 +2889,7 @@#define PCI_DEVICE_ID_INTEL_HDA_ICH8 0x284b#define PCI_DEVICE_ID_INTEL_ICH8_6 0x2850#define PCI_DEVICE_ID_INTEL_VMD_28C0 0x28c0 +#define PCI_DEVICE_ID_INTEL_VMD_28C1 0x28c1#define PCI_DEVICE_ID_INTEL_ICH9_0 0x2910#define PCI_DEVICE_ID_INTEL_ICH9_2 0x2912#define PCI_DEVICE_ID_INTEL_ICH9_3 0x2913
Signature · 1 line
2.52.0
[email protected]
AI reviewThank you for your contribution! Sashiko AI review found 4 potential issue(s) to consider:
Thank you for your contribution! Sashiko AI review found 4 potential issue(s) to consider:
New issues:
- [High] vmd_resume() unconditionally modifies legacy MSI remapping configuration on unsupported hardware.
Pre-existing issues:
- [High] SRCU structures are permanently leaked on probe error paths.
- [High] SRCU structs are destroyed while the managed IRQ handler is still registered, leading to a Use-After-Free.
- [High] Truncation of 64-bit physical addresses on 32-bit systems due to incorrect use of PCI_BASE_ADDRESS_MEM_MASK. --
commit f05a884975712ad2dc4d55677318e1620e041df9
Author: Nirmal Patel <[email protected]>
PCI: vmd: Add feature to scan BIOS enumerated devices.
Newer VMD with device ID 0x28c1 has unique settings compared to its predecessor where BIOS enumerates the entire VMD device tree and assigns respective configurations.
Quoted 4 lines
drivers/pci/controller/vmd.c +0
-0
[ ... ]
Quoted 8 lines
drivers/pci/controller/vmd.c +4
-0@@ -661,6 +684,46 @@ static int vmd_get_bus_number_start(struct vmd_dev *vmd)return 0 ; } +static int vmd_get_bus_info_from_bar4 (struct vmd_dev *vmd, +resource_size_t *offset1, +resource_size_t *offset2) +{
[ ... ]
drivers/pci/controller/vmd.c static int vmd_get_bus_number_start(struct vmd_dev *vmd) · line 721
Quoted 7 lines
drivers/pci/controller/vmd.c +7
-0+/* Calculate offsets like vmd_get_phys_offsets() does. */ +if (phys1) + *offset1 = vmd->dev->resource[VMD_MEMBAR1 ].start - + (phys1 &PCI_BASE_ADDRESS_MEM_MASK ); +if (phys2) + *offset2 = vmd->dev->resource[VMD_MEMBAR2 ].start - + (phys2 &PCI_BASE_ADDRESS_MEM_MASK );
[Severity: High]
Will this truncate the 64-bit physical addresses on 32-bit systems?
Since PCI_BASE_ADDRESS_MEM_MASK is ~0x0fUL, on 32-bit systems it evaluates to a 32-bit 0xFFFFFFF0. When bitwise ANDed with a 64-bit u64 like phys1 or phys2, the mask is zero-extended to 0x00000000FFFFFFF0ULL, clearing the upper 32 bits of the address.
Could this result in entirely incorrect memory mapping offsets for VMD devices mapped above 4GB?
This is a pre-existing issue in vmd_get_phys_offsets() as well, but it seems to be repeated here.
drivers/pci/controller/vmd.c static int vmd_alloc_irqs(struct vmd_dev *vmd) · line 775
Quoted 11 lines
drivers/pci/controller/vmd.c +5
-0+ +pci_iounmap (vmd->dev, bar4); +return 0 ; +} +static irqreturn_t vmd_irq (int irq,void *data) {struct vmd_irq_list *irqs = data;@@ -711,6 +774,53 @@ static int vmd_alloc_irqs(struct vmd_dev *vmd)return 0 ; }
[Severity: High]
This isn't a bug introduced by this patch, but does the driver leak srcu_struct structures on the probe error paths?
Looking at vmd_alloc_irqs(), it initializes srcu_struct structures in a loop. If a subsequent step fails (such as init_srcu_struct or vmd_create_irq_domain later in probe), vmd_probe() jumps to out_release_instance and returns an error without calling cleanup_srcu_struct() for the already initialized structures.
While IRQ vectors are automatically freed via devres, the srcu_struct allocations inside the per-CPU memory appear to be permanently leaked.
[Severity: High]
This is also a pre-existing issue, but does vmd_remove() introduce a use-after-free by manually destroying srcu_struct structures while the managed IRQ handler is still registered?
The driver registers its interrupt handlers using devm_request_irq(). Because devres handlers run only after vmd_remove() returns, the IRQ remains active and can fire during or after the srcu_struct structures are manually destroyed.
If vmd_irq() fires during this window, it calls srcu_read_lock() on a destroyed srcu_struct. Can this sequence trigger a use-after-free?
[ ... ]
Quoted 8 lines
drivers/pci/controller/vmd.c +0
-4@@ -868,19 +956,21 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features)* acceptable because the guest is usually CPU-limited and MSI * remapping doesn't become a performance bottleneck. */ -if (!(features &VMD_FEAT_CAN_BYPASS_MSI_REMAP ) || - offset[0 ] || offset[1 ]) { - ret =vmd_alloc_irqs (vmd); -if (ret)
drivers/pci/controller/vmd.c static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) · line 966
Quoted 10 lines
drivers/pci/controller/vmd.c +7
-2-return ret; +if (!(features &VMD_FEAT_USE_BIOS_INFO )) { +if (!(features &VMD_FEAT_CAN_BYPASS_MSI_REMAP ) || + offset[0 ] || offset[1 ]) { + ret =vmd_alloc_irqs (vmd); +if (ret) +return ret; -vmd_set_msi_remapping (vmd, true); +vmd_set_msi_remapping (vmd, true);
[Severity: High]
This patch bypasses MSI remapping initialization for VMD_FEAT_USE_BIOS_INFO in vmd_enable_domain(), but is it also necessary to update vmd_resume()?
vmd_resume() unconditionally calls vmd_set_msi_remapping(), which will read and write the legacy PCI_REG_VMCONFIG register during system resume. Could this cause undefined behavior or PCI aborts on the new hardware?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Nirmal Patel
maintainerQuoted 6 lines · On Thu, 21 May 2026 23:52:56 +0000 [email protected] wrote:
Thank you for your contribution! Sashiko AI review found 4 potential issue(s) to consider:
New issues:
- [High]vmd_resume()unconditionally modifies legacy MSI remapping configuration on unsupported hardware.
Ack.
Quoted 7 lines
Pre-existing issues:
- [High] SRCU structures are permanently leaked on probe error paths.
- [High] SRCU structs are destroyed while the managed IRQ handler is still registered, leading to a Use-After-Free.
- [High] Truncation of 64-bit physical addresses on 32-bit systems due to incorrect use ofPCI_BASE_ADDRESS_MEM_MASK. --
Proposing to fix these issues in separate patches to keep the patch size more manageable.
Quoted 111 lines
commit
f05a884975712ad2dc4d55677318e1620e041df9
Author: Nirmal Patel <[email protected]>PCI: vmd: Add feature to scan BIOS enumerated devices.
Newer VMD with device ID
0x28c1has unique settings compared to its predecessor where BIOS enumerates the entire VMD device tree and assigns respective configurations.Quoted 4 lines
(quoted diff) +0
-0
b/drivers/pci/controller/vmd.cindexd4ae250d4bc6d..15a3f6a357889100644 ---a/drivers/pci/controller/vmd.c
+++b/drivers/pci/controller/vmd.c[ ... ]
Quoted 8 lines
drivers/pci/controller/vmd.c +4
-0@@ -661,6 +684,46 @@ static int vmd_get_bus_number_start(struct vmd_dev *vmd) return 0; } +static int vmd_get_bus_info_from_bar4 (struct vmd_dev *vmd, +resource_size_t *offset1, +resource_size_t *offset2) +{[ ... ]
drivers/pci/controller/vmd.c static int vmd_get_bus_number_start(struct vmd_dev *vmd) · line 721
Quoted 7 lines
drivers/pci/controller/vmd.c +7
-0+/* Calculate offsets like vmd_get_phys_offsets() does. */ +if (phys1) + *offset1 = vmd->dev->resource[VMD_MEMBAR1 ].start - + (phys1 &PCI_BASE_ADDRESS_MEM_MASK ); +if (phys2) + *offset2 = vmd->dev->resource[VMD_MEMBAR2 ].start - + (phys2 &PCI_BASE_ADDRESS_MEM_MASK );[Severity: High]
Will this truncate the 64-bit physical addresses on 32-bit systems?Since
PCI_BASE_ADDRESS_MEM_MASKis ~0x0fUL, on 32-bit systems it evaluates to a 32-bit0xFFFFFFF0. When bitwise ANDed with a 64-bit u64 like phys1 or phys2, the mask is zero-extended to0x00000000FFFFFFF0ULL, clearing the upper 32 bits of the address.Could this result in entirely incorrect memory mapping offsets for VMD devices mapped above 4GB?
This is a pre-existing issue in
vmd_get_phys_offsets()as well, but it seems to be repeated here.drivers/pci/controller/vmd.c static int vmd_alloc_irqs(struct vmd_dev *vmd) · line 775
Quoted 11 lines
drivers/pci/controller/vmd.c +5
-0+ +pci_iounmap (vmd->dev, bar4); +return 0 ; +} +static irqreturn_t vmd_irq (int irq,void *data) {struct vmd_irq_list *irqs = data;@@ -711,6 +774,53 @@ static int vmd_alloc_irqs(struct vmd_dev *vmd)return 0 ; }[Severity: High]
This isn't a bug introduced by this patch, but does the driver leaksrcu_structstructures on the probe error paths?Looking at
vmd_alloc_irqs(), it initializessrcu_structstructures in a loop. If a subsequent step fails (such asinit_srcu_structorvmd_create_irq_domainlater in probe),vmd_probe()jumps toout_release_instanceand returns an error without callingcleanup_srcu_struct()for the already initialized structures.While IRQ vectors are automatically freed via devres, the
srcu_structallocations inside the per-CPU memory appear to be permanently leaked.[Severity: High]
This is also a pre-existing issue, but doesvmd_remove()introduce a use-after-free by manually destroyingsrcu_structstructures while the managed IRQ handler is still registered?The driver registers its interrupt handlers using
devm_request_irq(). Because devres handlers run only aftervmd_remove()returns, the IRQ remains active and can fire during or after thesrcu_structstructures are manually destroyed.If
vmd_irq()fires during this window, it callssrcu_read_lock()on a destroyedsrcu_struct. Can this sequence trigger a use-after-free?[ ... ]
Quoted 10 lines
(quoted diff) +0
-0@@ -868,19 +956,21 @@ static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) * acceptable because the guest is usually CPU-limited andMSI
drivers/pci/controller/vmd.c +0
-4* remapping doesn't become a performance bottleneck. */ -if (!(features &VMD_FEAT_CAN_BYPASS_MSI_REMAP ) || - offset[0 ] || offset[1 ]) { - ret =vmd_alloc_irqs (vmd); -if (ret)drivers/pci/controller/vmd.c static int vmd_enable_domain(struct vmd_dev *vmd, unsigned long features) · line 966
Quoted 10 lines
drivers/pci/controller/vmd.c +7
-2-return ret; +if (!(features &VMD_FEAT_USE_BIOS_INFO )) { +if (!(features &VMD_FEAT_CAN_BYPASS_MSI_REMAP ) || + offset[0 ] || offset[1 ]) { + ret =vmd_alloc_irqs (vmd); +if (ret) +return ret; -vmd_set_msi_remapping (vmd, true); +vmd_set_msi_remapping (vmd, true);[Severity: High]
This patch bypasses MSI remapping initialization forVMD_FEAT_USE_BIOS_INFOinvmd_enable_domain(), but is it also necessary to updatevmd_resume()?
Quoted 5 lines
vmd_resume()unconditionally callsvmd_set_msi_remapping(), which will read and write the legacyPCI_REG_VMCONFIGregister during system resume. Could this cause undefined behavior or PCI aborts on the new hardware?
I will address this issue. We dont want vmd_resume to ser msi for unsupported devices.