Skip to contentSkip to conversation

xen/pci: detect when BARs overlap RAM

RFC PATCH In review Roger Pau Monne 3 participants 11 messages Jan 26, 2022 – Jan 31, 2022

lore
Share & tools
lore
#0 lore raw

RFC PATCH 1/1 xen/pci: detect when BARs overlap RAM

One of the boxes where I was attempting to boot Xen in PVH dom0 mode has quirky firmware, as it will handover with a device with memory decoding enabled and a BAR of size 4K at address 0. Such BAR overlaps with a RAM range on the e820.

This interacts badly with the dom0 PVH build, as BARs will be setup on the p2m before RAM, so if there's a BAR positioned over a RAM region it will trigger a domain crash when the dom0 builder attempts to populate that region with a regular RAM page.

It's in general a very bad idea to have a BAR overlapping with a RAM region, so add some sanity checks for devices that are added with memory decoding enabled in order to assure that BARs are not placed on top of memory regions. If overlaps are detected just disable the memory decoding bit for the device and expect the hardware domain to properly position the BAR.

---
RFC because:
- Not sure the best way to implement is_iomem_range on Arm. BARs can be quite big, so iterating over every possible page is not ideal.
- is_iomem_page cannot be used for this purpose on x86, because all the low 1MB will return true due to belonging to dom_io.
- VF BARs are not checked. Should we also check them and disable VF if overlaps in a followup patch?

---
 xen/arch/arm/mm.c             | 11 ++++++
 xen/arch/x86/mm.c             | 20 +++++++++++
 xen/drivers/passthrough/pci.c | 68 ++++++++++++++++++++++++++++++++++-
 xen/include/xen/mm.h          |  2 ++
 4 files changed, 100 insertions(+), 1 deletion(-)
xen/arch/arm/mm.c +11 -0
index eea926d823..fa4cee64c7 100644
@@ -1625,6 +1625,17 @@ bool is_iomem_page(mfn_t mfn)
     return !mfn_valid(mfn);
 }
 
+bool is_iomem_range(uint64_t start, uint64_t size)
+{
+    unsigned int i;
+
+    for ( i = 0; i < PFN_UP(size); i++ )
+        if ( !is_iomem_page(_mfn(PFN_DOWN(start) + i)) )
+            return false;
+
+    return true;
+}
+
 void clear_and_clean_page(struct page_info *page)
 {
     void *p = __map_domain_page(page);
xen/arch/x86/mm.c +20 -0
index 1397f83e41..03699b2227 100644
@@ -479,6 +479,26 @@ unsigned int page_get_ram_type(mfn_t mfn)
     return type ?: RAM_TYPE_UNKNOWN;
 }
 
+bool is_iomem_range(uint64_t start, uint64_t size)
+{
+    unsigned int i;
+
+    for ( i = 0; i < e820.nr_map; i++ )
+    {
+        struct e820entry *entry = &e820.map[i];
+
+        if ( entry->type != E820_RAM && entry->type != E820_ACPI &&
+             entry->type != E820_NVS )
+            continue;
+
+        if ( start < entry->addr + entry->size &&
+             entry->addr < start + size )
+            return false;
+    }
+
+    return true;
+}
+
 unsigned long domain_get_maximum_gpfn(struct domain *d)
 {
     if ( is_hvm_domain(d) )
xen/drivers/passthrough/pci.c +67 -1

98 lines not on this page · raw message

xen/include/xen/mm.h +2 -0
index 5db26ed477..764dcad5b3 100644
@@ -554,6 +554,8 @@ int __must_check steal_page(struct domain *d, struct page_info *page,
 int page_is_ram_type(unsigned long mfn, unsigned long mem_type);
 /* Returns the page type(s). */
 unsigned int page_get_ram_type(mfn_t mfn);
+/* Check if a range is in IO suitable memory. */
+bool is_iomem_range(uint64_t start, uint64_t size);
 
 /* Prepare/destroy a ring for a dom0 helper. Helper with talk
  * with Xen on behalf of this domain. */
Signature · 1 line
2.34.1
#1 lore raw
Quoted 5 lines · Roger Pau Monne

RFC because:
- Not sure the best way to implement is_iomem_range on Arm. BARs can be quite big, so iterating over every possible page is not ideal.
- is_iomem_page cannot be used for this purpose on x86, because all the low 1MB will return true due to belonging to dom_io.

I don't see an issue there - if you were to us it, you'd end up with the same scalability issue as you point out for Arm.

Quoted 2 lines

- VF BARs are not checked. Should we also check them and disable VF if overlaps in a followup patch?

Not sure about "followup patch", but once we support SR-IOV for PVH, that'll want checking, yes.

xen/arch/x86/mm.c unsigned int page_get_ram_type(mfn_t mfn) · line 482

Quoted 7 lines
xen/arch/x86/mm.c +1 -0
@@ -479,6 +479,26 @@ unsigned int page_get_ram_type(mfn_t mfn)
     return type ?: RAM_TYPE_UNKNOWN;
 }
 
+bool is_iomem_range(uint64_t start, uint64_t size)

Might be nice to have this sit next it is_iomem_page(). And wouldn't at least "start" better be paddr_t?

xen/arch/x86/mm.c unsigned int page_get_ram_type(mfn_t mfn) · line 488

Quoted 6 lines
xen/arch/x86/mm.c +6 -0
+{
+    unsigned int i;
+
+    for ( i = 0; i < e820.nr_map; i++ )
+    {
+        struct e820entry *entry = &e820.map[i];

const?

xen/arch/x86/mm.c unsigned int page_get_ram_type(mfn_t mfn) · line 492

Quoted 3 lines
xen/arch/x86/mm.c +3 -0
+        if ( entry->type != E820_RAM && entry->type != E820_ACPI &&
+             entry->type != E820_NVS )
+            continue;

I think UNUSABLE also needs checking for here. Even further, it might be best to reject any range overlapping an E820 entry, since colliding with a RESERVED one could mean an overlap with e.g. MMCFG space.

Quoted 25 lines
xen/arch/x86/mm.c +16 -0
@@ -267,11 +270,74 @@ static void check_pdev(const struct pci_dev *pdev)
         }
         break;
 
+    case PCI_HEADER_TYPE_NORMAL:
+        nbars = PCI_HEADER_NORMAL_NR_BARS;
+        rom_pos = PCI_ROM_ADDRESS;
+        break;
+
     case PCI_HEADER_TYPE_CARDBUS:
         /* TODO */
         break;
     }
 #undef PCI_STATUS_CHECK
+
+    /* Check if BARs overlap with RAM regions. */
+    val = pci_conf_read16(pdev->sbdf, PCI_COMMAND);
+    if ( !(val & PCI_COMMAND_MEMORY) || pdev->ignore_bars )
+        return;
+
+    pci_conf_write16(pdev->sbdf, PCI_COMMAND, val & ~PCI_COMMAND_MEMORY);
+    for ( i = 0; i < nbars; )
+    {
+        uint64_t addr, size;
+        unsigned int reg = PCI_BASE_ADDRESS_0 + i * 4;

xen/arch/x86/mm.c static void check_pdev(const struct pci_dev *pdev) · line 303

Quoted 10 lines
xen/arch/x86/mm.c +10 -0
+        int rc = 1;
+
+        if ( (pci_conf_read32(pdev->sbdf, reg) & PCI_BASE_ADDRESS_SPACE) !=
+             PCI_BASE_ADDRESS_SPACE_MEMORY )
+            goto next;
+
+        rc = pci_size_mem_bar(pdev->sbdf, reg, &addr, &size,
+                              (i == nbars - 1) ? PCI_BAR_LAST : 0);
+        if ( rc < 0 )
+            return;

This isn't very nice, but probably the best you can do. Might be worth a comment, though.

xen/arch/x86/mm.c static void check_pdev(const struct pci_dev *pdev) · line 312

Quoted 9 lines
xen/arch/x86/mm.c +9 -0
+        if ( size && !is_iomem_range(addr, size) )
+        {
+            /*
+             * Return without enabling memory decoding if BAR overlaps with
+             * RAM region.
+             */
+            printk(XENLOG_WARNING
+                   "%pp: disabled: BAR%u [%" PRIx64 ", %" PRIx64
+                   ") overlaps with RAM\n",

Mentioning "RAM" in comment and log message is potentially misleading when considering what other types get also checked (irrespective of my earlier comment). (Ftaod I don't mind the title using "RAM".)

Also please don't wrap the format string (also again for ROM below).

xen/arch/x86/mm.c static struct pci_dev *alloc_pdev(struct pci_seg *pseg, u8 bus, u8 devfn) · line 469

Quoted 7 lines
xen/arch/x86/mm.c +1 -1
@@ -399,8 +465,8 @@ static struct pci_dev *alloc_pdev(struct pci_seg *pseg, u8 bus, u8 devfn)
             break;
     }
 
-    check_pdev(pdev);
     apply_quirks(pdev);
+    check_pdev(pdev);

I can't find the description say anything about this re-ordering. What's the deal here?

Jan

#2 lore raw
Quoted 22 lines · Jan Beulich
Quoted 5 lines · Roger Pau Monne

RFC because:
- Not sure the best way to implement is_iomem_range on Arm. BARs can be quite big, so iterating over every possible page is not ideal.
- is_iomem_page cannot be used for this purpose on x86, because all the low 1MB will return true due to belonging to dom_io.

I don't see an issue there - if you were to us it, you'd end up with the same scalability issue as you point out for Arm.

Quoted 2 lines

- VF BARs are not checked. Should we also check them and disable VF if overlaps in a followup patch?

Not sure about "followup patch", but once we support SR-IOV for PVH, that'll want checking, yes.

Quoted 6 lines
xen/arch/x86/mm.c +0 -0
@@ -479,6 +479,26 @@ unsigned int page_get_ram_type(mfn_t mfn)
     return type ?: RAM_TYPE_UNKNOWN;
 }
 
Quoted 4 lines

xen/arch/x86/mm.c unsigned int page_get_ram_type(mfn_t mfn) · line 482

Quoted 1 line
xen/arch/x86/mm.c +1 -0
+bool is_iomem_range(uint64_t start, uint64_t size)

Might be nice to have this sit next it is_iomem_page(). And wouldn't at least "start" better be paddr_t?

(paddr_t, size_t) would be better, but AFAICT size_t can be an unsigned long and on Arm32 that won't be suitable for holding a 64bit BAR size.

Quoted 9 lines

xen/arch/x86/mm.c unsigned int page_get_ram_type(mfn_t mfn) · line 488

Quoted 6 lines
xen/arch/x86/mm.c +6 -0
+{
+    unsigned int i;
+
+    for ( i = 0; i < e820.nr_map; i++ )
+    {
+        struct e820entry *entry = &e820.map[i];

const?

Quoted 7 lines

xen/arch/x86/mm.c unsigned int page_get_ram_type(mfn_t mfn) · line 492

Quoted 3 lines
xen/arch/x86/mm.c +3 -0
+        if ( entry->type != E820_RAM && entry->type != E820_ACPI &&
+             entry->type != E820_NVS )
+            continue;

I think UNUSABLE also needs checking for here. Even further, it might be best to reject any range overlapping an E820 entry, since colliding with a RESERVED one could mean an overlap with e.g. MMCFG space.

Hm, I've though about adding UNUSABLE and RESERVED (and should have added a note about this), but that might be too restrictive.

Quoted 49 lines
Quoted 25 lines
xen/arch/x86/mm.c +16 -0
@@ -267,11 +270,74 @@ static void check_pdev(const struct pci_dev *pdev)
         }
         break;
 
+    case PCI_HEADER_TYPE_NORMAL:
+        nbars = PCI_HEADER_NORMAL_NR_BARS;
+        rom_pos = PCI_ROM_ADDRESS;
+        break;
+
     case PCI_HEADER_TYPE_CARDBUS:
         /* TODO */
         break;
     }
 #undef PCI_STATUS_CHECK
+
+    /* Check if BARs overlap with RAM regions. */
+    val = pci_conf_read16(pdev->sbdf, PCI_COMMAND);
+    if ( !(val & PCI_COMMAND_MEMORY) || pdev->ignore_bars )
+        return;
+
+    pci_conf_write16(pdev->sbdf, PCI_COMMAND, val & ~PCI_COMMAND_MEMORY);
+    for ( i = 0; i < nbars; )
+    {
+        uint64_t addr, size;
+        unsigned int reg = PCI_BASE_ADDRESS_0 + i * 4;

xen/arch/x86/mm.c static void check_pdev(const struct pci_dev *pdev) · line 303

Quoted 10 lines
xen/arch/x86/mm.c +10 -0
+        int rc = 1;
+
+        if ( (pci_conf_read32(pdev->sbdf, reg) & PCI_BASE_ADDRESS_SPACE) !=
+             PCI_BASE_ADDRESS_SPACE_MEMORY )
+            goto next;
+
+        rc = pci_size_mem_bar(pdev->sbdf, reg, &addr, &size,
+                              (i == nbars - 1) ? PCI_BAR_LAST : 0);
+        if ( rc < 0 )
+            return;

This isn't very nice, but probably the best you can do. Might be worth a comment, though.

xen/arch/x86/mm.c static void check_pdev(const struct pci_dev *pdev) · line 312

Quoted 9 lines
xen/arch/x86/mm.c +9 -0
+        if ( size && !is_iomem_range(addr, size) )
+        {
+            /*
+             * Return without enabling memory decoding if BAR overlaps with
+             * RAM region.
+             */
+            printk(XENLOG_WARNING
+                   "%pp: disabled: BAR%u [%" PRIx64 ", %" PRIx64
+                   ") overlaps with RAM\n",
Quoted 3 lines

Mentioning "RAM" in comment and log message is potentially misleading when considering what other types get also checked (irrespective of my earlier comment). (Ftaod I don't mind the title using "RAM".)

Would the message below be better?

"%pp disabled: BAR%u [%" PRIx64 ", %" PRIx64 ") overlap detected\n"

Quoted 10 lines

xen/arch/x86/mm.c static struct pci_dev *alloc_pdev(struct pci_seg *pseg, u8 bus, u8 devfn) · line 469

Quoted 7 lines
xen/arch/x86/mm.c +1 -1
@@ -399,8 +465,8 @@ static struct pci_dev *alloc_pdev(struct pci_seg *pseg, u8 bus, u8 devfn)
             break;
     }
 
-    check_pdev(pdev);
     apply_quirks(pdev);
+    check_pdev(pdev);

I can't find the description say anything about this re-ordering. What's the deal here?

Should have mentioned: this is required so that ignore_bars is set prior to calling check_pdev.

Thanks, Roger.

#3 lore raw
Quoted 8 lines · Roger Pau Monné
Quoted 7 lines · Jan Beulich
Quoted 6 lines · Roger Pau Monne
xen/arch/x86/mm.c +0 -0
@@ -479,6 +479,26 @@ unsigned int page_get_ram_type(mfn_t mfn)
     return type ?: RAM_TYPE_UNKNOWN;
 }
 
Quoted 8 lines
Quoted 4 lines

xen/arch/x86/mm.c unsigned int page_get_ram_type(mfn_t mfn) · line 482

Quoted 1 line
xen/arch/x86/mm.c +1 -0
+bool is_iomem_range(uint64_t start, uint64_t size)

Might be nice to have this sit next it is_iomem_page(). And wouldn't at least "start" better be paddr_t?

(paddr_t, size_t) would be better, but AFAICT size_t can be an unsigned long and on Arm32 that won't be suitable for holding a 64bit BAR size.

Indeed. We'd need a resource_size_t or alike.

Quoted 13 lines
Quoted 13 lines

xen/arch/x86/mm.c unsigned int page_get_ram_type(mfn_t mfn) · line 488

Quoted 6 lines
xen/arch/x86/mm.c +6 -0
+{
+    unsigned int i;
+
+    for ( i = 0; i < e820.nr_map; i++ )
+    {
+        struct e820entry *entry = &e820.map[i];

const?

Quoted 3 lines
xen/arch/x86/mm.c +3 -0
+        if ( entry->type != E820_RAM && entry->type != E820_ACPI &&
+             entry->type != E820_NVS )
+            continue;
Quoted 6 lines
Quoted 3 lines

I think UNUSABLE also needs checking for here. Even further, it might be best to reject any range overlapping an E820 entry, since colliding with a RESERVED one could mean an overlap with e.g. MMCFG space.

Hm, I've though about adding UNUSABLE and RESERVED (and should have added a note about this), but that might be too restrictive.

Why (other than because of firmware bugs)?

Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
xen/arch/x86/mm.c +9 -0
+        if ( size && !is_iomem_range(addr, size) )
+        {
+            /*
+             * Return without enabling memory decoding if BAR overlaps with
+             * RAM region.
+             */
+            printk(XENLOG_WARNING
+                   "%pp: disabled: BAR%u [%" PRIx64 ", %" PRIx64
+                   ") overlaps with RAM\n",
Quoted 7 lines
Quoted 3 lines

Mentioning "RAM" in comment and log message is potentially misleading when considering what other types get also checked (irrespective of my earlier comment). (Ftaod I don't mind the title using "RAM".)

Would the message below be better?

"%pp disabled: BAR%u [%" PRIx64 ", %" PRIx64 ") overlap detected\n"

This or maybe

"%pp disabled: BAR%u [%" PRIx64 ", %" PRIx64 ") overlaps with memory map\n"

in particular if, on x86, we'd be checking for any E820 entry type.

Quoted 11 lines
Quoted 10 lines

xen/arch/x86/mm.c static struct pci_dev *alloc_pdev(struct pci_seg *pseg, u8 bus, u8 devfn) · line 469

Quoted 7 lines
xen/arch/x86/mm.c +1 -1
@@ -399,8 +465,8 @@ static struct pci_dev *alloc_pdev(struct pci_seg *pseg, u8 bus, u8 devfn)
             break;
     }
 
-    check_pdev(pdev);
     apply_quirks(pdev);
+    check_pdev(pdev);

I can't find the description say anything about this re-ordering. What's the deal here?

Quoted 2 lines

Should have mentioned: this is required so that ignore_bars is set prior to calling check_pdev.

Ah, I see. Makes sense, but indeed wants mentioning.

Jan

#4 lore raw
Quoted 7 lines · Jan Beulich
Quoted 6 lines · Roger Pau Monné
Quoted 5 lines · Jan Beulich
Quoted 3 lines · Roger Pau Monne
xen/arch/x86/mm.c +3 -0
+        if ( entry->type != E820_RAM && entry->type != E820_ACPI &&
+             entry->type != E820_NVS )
+            continue;
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

I think UNUSABLE also needs checking for here. Even further, it might be best to reject any range overlapping an E820 entry, since colliding with a RESERVED one could mean an overlap with e.g. MMCFG space.

Hm, I've though about adding UNUSABLE and RESERVED (and should have added a note about this), but that might be too restrictive.

Why (other than because of firmware bugs)?

Handling overlaps with those wasn't strictly needed to solve the issue at hand, but I didn't take into account that an overlap with a MMCFG region would be equally bad. I've extended the check to cover any E820 entry.

Quoted 16 lines
Quoted 16 lines
Quoted 10 lines

xen/drivers/passthrough/pci.c static void check_pdev(const struct pci_dev *pdev) · line 312

Quoted 9 lines
xen/drivers/passthrough/pci.c +9 -0
+        if ( size && !is_iomem_range(addr, size) )
+        {
+            /*
+             * Return without enabling memory decoding if BAR overlaps with
+             * RAM region.
+             */
+            printk(XENLOG_WARNING
+                   "%pp: disabled: BAR%u [%" PRIx64 ", %" PRIx64
+                   ") overlaps with RAM\n",
Quoted 3 lines

Mentioning "RAM" in comment and log message is potentially misleading when considering what other types get also checked (irrespective of my earlier comment). (Ftaod I don't mind the title using "RAM".)

Would the message below be better?

Quoted 7 lines
Quoted 1 line

"%pp disabled: BAR%u [%" PRIx64 ", %" PRIx64 ") overlap detected\n"

This or maybe

"%pp disabled: BAR%u [%" PRIx64 ", %" PRIx64 ") overlaps with memory map\n"

in particular if, on x86, we'd be checking for any E820 entry type.

Done.

Thanks, Roger.

#5 lore raw
In reply to Roger Pau Monné
Quoted 8 lines · Roger Pau Monné
Quoted 7 lines · Jan Beulich
Quoted 6 lines · Roger Pau Monne
xen/arch/x86/mm.c +0 -0
@@ -479,6 +479,26 @@ unsigned int page_get_ram_type(mfn_t mfn)
     return type ?: RAM_TYPE_UNKNOWN;
 }
 
Quoted 8 lines
Quoted 4 lines

xen/arch/x86/mm.c unsigned int page_get_ram_type(mfn_t mfn) · line 482

Quoted 1 line
xen/arch/x86/mm.c +1 -0
+bool is_iomem_range(uint64_t start, uint64_t size)

Might be nice to have this sit next it is_iomem_page(). And wouldn't at least "start" better be paddr_t?

(paddr_t, size_t) would be better, but AFAICT size_t can be an unsigned long and on Arm32 that won't be suitable for holding a 64bit BAR size.

Talking of representing the range - a BAR occupying one part of a page overlapping an E820 entry covering another part of that same page is going to be equally bad, I think. The range may therefore want expressing in page granularity. This may then be easier as [start,end], at least on the calling side (can use PFN_DOWN() twice then).

Jan

#6 lore raw
Quoted 18 lines · Jan Beulich
Quoted 8 lines · Roger Pau Monné
Quoted 7 lines · Jan Beulich
Quoted 6 lines · Roger Pau Monne
xen/arch/x86/mm.c +0 -0
@@ -479,6 +479,26 @@ unsigned int page_get_ram_type(mfn_t mfn)
     return type ?: RAM_TYPE_UNKNOWN;
 }
 
Quoted 8 lines
Quoted 4 lines

xen/arch/x86/mm.c unsigned int page_get_ram_type(mfn_t mfn) · line 482

Quoted 1 line
xen/arch/x86/mm.c +1 -0
+bool is_iomem_range(uint64_t start, uint64_t size)

Might be nice to have this sit next it is_iomem_page(). And wouldn't at least "start" better be paddr_t?

(paddr_t, size_t) would be better, but AFAICT size_t can be an unsigned long and on Arm32 that won't be suitable for holding a 64bit BAR size.

Quoted 6 lines

Talking of representing the range - a BAR occupying one part of a page overlapping an E820 entry covering another part of that same page is going to be equally bad, I think. The range may therefore want expressing in page granularity. This may then be easier as [start,end], at least on the calling side (can use PFN_DOWN() twice then).

Yes, indeed, would likely be better. Then I can make both parameters unsigned long.

I also think is_memory_hole would be a suitable name given the new implementation?

Thanks, Roger.

#7 lore raw
Quoted 26 lines · Roger Pau Monné
Quoted 18 lines · Jan Beulich
Quoted 8 lines · Roger Pau Monné
Quoted 7 lines · Jan Beulich
Quoted 6 lines · Roger Pau Monne
xen/arch/x86/mm.c +0 -0
@@ -479,6 +479,26 @@ unsigned int page_get_ram_type(mfn_t mfn)
     return type ?: RAM_TYPE_UNKNOWN;
 }
 
Quoted 8 lines
Quoted 4 lines

xen/arch/x86/mm.c unsigned int page_get_ram_type(mfn_t mfn) · line 482

Quoted 1 line
xen/arch/x86/mm.c +1 -0
+bool is_iomem_range(uint64_t start, uint64_t size)

Might be nice to have this sit next it is_iomem_page(). And wouldn't at least "start" better be paddr_t?

(paddr_t, size_t) would be better, but AFAICT size_t can be an unsigned long and on Arm32 that won't be suitable for holding a 64bit BAR size.

Quoted 6 lines

Talking of representing the range - a BAR occupying one part of a page overlapping an E820 entry covering another part of that same page is going to be equally bad, I think. The range may therefore want expressing in page granularity. This may then be easier as [start,end], at least on the calling side (can use PFN_DOWN() twice then).

Quoted 5 lines

Yes, indeed, would likely be better. Then I can make both parameters unsigned long.

I also think is_memory_hole would be a suitable name given the new implementation?

I'd be fine with this name, fwiw.

Jan

#8 lore raw
In reply to Roger Pau Monne
Quoted 10 lines · Roger Pau Monne

One of the boxes where I was attempting to boot Xen in PVH dom0 mode has quirky firmware, as it will handover with a device with memory decoding enabled and a BAR of size 4K at address 0. Such BAR overlaps with a RAM range on the e820.

This interacts badly with the dom0 PVH build, as BARs will be setup on the p2m before RAM, so if there's a BAR positioned over a RAM region it will trigger a domain crash when the dom0 builder attempts to populate that region with a regular RAM page.

Quoted 8 lines

It's in general a very bad idea to have a BAR overlapping with a RAM region, so add some sanity checks for devices that are added with memory decoding enabled in order to assure that BARs are not placed on top of memory regions. If overlaps are detected just disable the memory decoding bit for the device and expect the hardware domain to properly position the BAR.

I'm not sure this is a sensible approach.

A bar at 0 is utterly unusable, because it is outside of the host bridge window.

The absence of any coherent GPA map for guests (needs fixing for oh so many reasons) is a primary contributor to the problem, because Xen *should* know where the guest's low and high MMIO windows are before attempting to map BARs into the guest.

The proper fix is to teach Xen about GPA maps, and reject BAR insertion outside of the respective bridge windows.

An fix in the short term would be to disable the problematic BAR when scanning the PCI bus to being with.

~Andrew

#9 lore raw
Quoted 22 lines · Andrew Cooper
Quoted 10 lines · Roger Pau Monne

One of the boxes where I was attempting to boot Xen in PVH dom0 mode has quirky firmware, as it will handover with a device with memory decoding enabled and a BAR of size 4K at address 0. Such BAR overlaps with a RAM range on the e820.

This interacts badly with the dom0 PVH build, as BARs will be setup on the p2m before RAM, so if there's a BAR positioned over a RAM region it will trigger a domain crash when the dom0 builder attempts to populate that region with a regular RAM page.

Quoted 8 lines

It's in general a very bad idea to have a BAR overlapping with a RAM region, so add some sanity checks for devices that are added with memory decoding enabled in order to assure that BARs are not placed on top of memory regions. If overlaps are detected just disable the memory decoding bit for the device and expect the hardware domain to properly position the BAR.

I'm not sure this is a sensible approach.

Quoted 7 lines

A bar at 0 is utterly unusable, because it is outside of the host bridge window.

The absence of any coherent GPA map for guests (needs fixing for oh so many reasons) is a primary contributor to the problem, because Xen *should* know where the guest's low and high MMIO windows are before attempting to map BARs into the guest.

But this is all about Dom0, ...

Quoted 2 lines

The proper fix is to teach Xen about GPA maps, and reject BAR insertion outside of the respective bridge windows.

... and hence this isn't about "insertion", but about what we find upon booting.

Quoted 2 lines

An fix in the short term would be to disable the problematic BAR when scanning the PCI bus to being with.

You can't disable an individual BAR (except for the ROM one), you need to disable all memory ones collectively (by disabling memory decode). Which is what Roger does.

Jan

#10 lore raw
Quoted 39 lines · Jan Beulich
Quoted 22 lines · Andrew Cooper
Quoted 10 lines · Roger Pau Monne

One of the boxes where I was attempting to boot Xen in PVH dom0 mode has quirky firmware, as it will handover with a device with memory decoding enabled and a BAR of size 4K at address 0. Such BAR overlaps with a RAM range on the e820.

This interacts badly with the dom0 PVH build, as BARs will be setup on the p2m before RAM, so if there's a BAR positioned over a RAM region it will trigger a domain crash when the dom0 builder attempts to populate that region with a regular RAM page.

Quoted 8 lines

It's in general a very bad idea to have a BAR overlapping with a RAM region, so add some sanity checks for devices that are added with memory decoding enabled in order to assure that BARs are not placed on top of memory regions. If overlaps are detected just disable the memory decoding bit for the device and expect the hardware domain to properly position the BAR.

I'm not sure this is a sensible approach.

Quoted 7 lines

A bar at 0 is utterly unusable, because it is outside of the host bridge window.

The absence of any coherent GPA map for guests (needs fixing for oh so many reasons) is a primary contributor to the problem, because Xen *should* know where the guest's low and high MMIO windows are before attempting to map BARs into the guest.

But this is all about Dom0, ...

Quoted 2 lines

The proper fix is to teach Xen about GPA maps, and reject BAR insertion outside of the respective bridge windows.

... and hence this isn't about "insertion", but about what we find upon booting.

Quoted 6 lines
Quoted 2 lines

An fix in the short term would be to disable the problematic BAR when scanning the PCI bus to being with.

You can't disable an individual BAR (except for the ROM one), you need to disable all memory ones collectively (by disabling memory decode). Which is what Roger does.

I've thought about just disabling the ROM BAR if that's the only conflicting one, but it was easier given the context to just disable memory decoding globally like it's done for plain BARs. Didn't seem worth the extra code to special case the ROM BAR disabling.

Thanks, Roger.