Skip to contentSkip to conversation

libxl: fix memory map reported to PVH guests

PATCH Superseded Roger Pau Monne 4 participants 10 messages Apr 20, 2018

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 libxl: fix memory map reported to PVH guests

PVH guests with 4GB of RAM or more get a memory map like the following:

0x00000000000000 - 0x000000fee00000 RAM
0x000000fee00000 - 0x00000100000000 RESERVED
0x000000fc009000 - 0x000000fc009040 ACPI
0x000000fc000000 - 0x000000fc001000 ACPI
0x000000fc001000 - 0x000000fc009000 ACPI
0x00000100000000 - 0x000001fb200400 RAM

This is wrong because ACPI regions are also reported as RAM regions. The cause of this issue is not setting a big enough MMIO hole, current libxl code only takes into account the address of the local APIC page and the reserved pages in order to set the size of the MMIO hole, when it should also take into account the location of the ACPI tables.

After the fix the layout reported for the same guest is:

0x00000000000000 - 0x000000fc000000 RAM
0x000000fc000000 - 0x00000100000000 RESERVED
0x000000fc009000 - 0x000000fc009040 ACPI
0x000000fc000000 - 0x000000fc001000 ACPI
0x000000fc001000 - 0x000000fc009000 ACPI
0x00000100000000 - 0x000001fe000400 RAM

---
Cc: Ian Jackson <[email protected]>
Cc: Wei Liu <[email protected]>
Cc: Juergen Gross <[email protected]>

---
 tools/libxl/libxl_arch.h     |  1 +
 tools/libxl/libxl_dom.c      | 20 +++++++++++---------
 tools/libxl/libxl_x86_acpi.c |  1 -
 3 files changed, 12 insertions(+), 10 deletions(-)
tools/libxl/libxl_arch.h +1 -0
index 318c111bb4..74a5af3cf3 100644
@@ -76,6 +76,7 @@ int libxl__arch_extra_memory(libxl__gc *gc,
 #if defined(__i386__) || defined(__x86_64__)
 
 #define LAPIC_BASE_ADDRESS  0xfee00000
+#define ACPI_INFO_PHYSICAL_ADDRESS 0xfc000000
 
 int libxl__dom_load_acpi(libxl__gc *gc,
                          const libxl_domain_build_info *b_info,
tools/libxl/libxl_dom.c +11 -9
index 8c3607b0f5..f0fd5fd3a3 100644
@@ -1229,15 +1229,17 @@ int libxl__build_hvm(libxl__gc *gc, uint32_t domid,
         dom->mmio_size = HVM_BELOW_4G_MMIO_LENGTH;
     else if (dom->mmio_size == 0 && !device_model) {
 #if defined(__i386__) || defined(__x86_64__)
-        if (libxl_defbool_val(info->apic)) {
-            /* Make sure LAPIC_BASE_ADDRESS is below special pages */
-            assert(((((X86_HVM_END_SPECIAL_REGION - X86_HVM_NR_SPECIAL_PAGES)
-                      << XC_PAGE_SHIFT) - LAPIC_BASE_ADDRESS)) >= XC_PAGE_SIZE);
-            dom->mmio_size = GB(4) - LAPIC_BASE_ADDRESS;
-        } else
-            dom->mmio_size = GB(4) -
-                ((X86_HVM_END_SPECIAL_REGION - X86_HVM_NR_SPECIAL_PAGES)
-                 << XC_PAGE_SHIFT);
+        /*
+         * Make sure the local APIC page, the ACPI tables and the special pages
+         * are inside the MMIO hole.
+         */
+        xen_paddr_t start =
+            (X86_HVM_END_SPECIAL_REGION - X86_HVM_NR_SPECIAL_PAGES) <<
+            XC_PAGE_SHIFT;
+
+        start = min_t(xen_paddr_t, start, LAPIC_BASE_ADDRESS);
+        start = min_t(xen_paddr_t, start, ACPI_INFO_PHYSICAL_ADDRESS);
+        dom->mmio_size = GB(4) - start;
 #else
         assert(1);
 #endif
tools/libxl/libxl_x86_acpi.c +0 -1
index 143ce66644..ed6610c84e 100644
@@ -22,7 +22,6 @@
 
  /* Number of pages holding ACPI tables */
 #define NUM_ACPI_PAGES 16
-#define ACPI_INFO_PHYSICAL_ADDRESS 0xfc000000
 
 struct libxl_acpi_ctxt {
     struct acpi_ctxt c;
Signature · 7 lines
2.17.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#1 lore raw
Quoted 18 lines · Roger Pau Monne

PVH guests with 4GB of RAM or more get a memory map like the following:

0x00000000000000 - 0x000000fee00000 RAM
0x000000fee00000 - 0x00000100000000 RESERVED
0x000000fc009000 - 0x000000fc009040 ACPI
0x000000fc000000 - 0x000000fc001000 ACPI
0x000000fc001000 - 0x000000fc009000 ACPI
0x00000100000000 - 0x000001fb200400 RAM

This is wrong because ACPI regions are also reported as RAM regions. The cause of this issue is not setting a big enough MMIO hole, current libxl code only takes into account the address of the local APIC page and the reserved pages in order to set the size of the MMIO hole, when it should also take into account the location of the ACPI tables.

After the fix the layout reported for the same guest is:

Quoted 8 lines

0x00000000000000 - 0x000000fc000000 RAM
0x000000fc000000 - 0x00000100000000 RESERVED
0x000000fc009000 - 0x000000fc009040 ACPI
0x000000fc000000 - 0x000000fc001000 ACPI
0x000000fc001000 - 0x000000fc009000 ACPI
0x00000100000000 - 0x000001fe000400 RAM

Juergen

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#2 lore raw
In reply to Roger Pau Monne
Quoted 17 lines
Quoted 1 line
Quoted 1 line · On 20.04.18 at 15:52, <[email protected]> wrote:

PVH guests with 4GB of RAM or more get a memory map like the following:

0x00000000000000 - 0x000000fee00000 RAM
0x000000fee00000 - 0x00000100000000 RESERVED
0x000000fc009000 - 0x000000fc009040 ACPI
0x000000fc000000 - 0x000000fc001000 ACPI
0x000000fc001000 - 0x000000fc009000 ACPI
0x00000100000000 - 0x000001fb200400 RAM

This is wrong because ACPI regions are also reported as RAM regions. The cause of this issue is not setting a big enough MMIO hole, current libxl code only takes into account the address of the local APIC page and the reserved pages in order to set the size of the MMIO hole, when it should also take into account the location of the ACPI tables.

Quoted 8 lines

After the fix the layout reported for the same guest is:

0x00000000000000 - 0x000000fc000000 RAM
0x000000fc000000 - 0x00000100000000 RESERVED
0x000000fc009000 - 0x000000fc009040 ACPI
0x000000fc000000 - 0x000000fc001000 ACPI
0x000000fc001000 - 0x000000fc009000 ACPI
0x00000100000000 - 0x000001fe000400 RAM

But this is still wrong - no two regions may overlap, regardless of type. Also you appear to be losing all RAM in [fc009000, fee00000).

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#3 lore raw
Quoted 19 lines · Jan Beulich
Quoted 19 lines
Quoted 1 line
Quoted 1 line · On 20.04.18 at 15:52, <[email protected]> wrote:

PVH guests with 4GB of RAM or more get a memory map like the following:

0x00000000000000 - 0x000000fee00000 RAM
0x000000fee00000 - 0x00000100000000 RESERVED
0x000000fc009000 - 0x000000fc009040 ACPI
0x000000fc000000 - 0x000000fc001000 ACPI
0x000000fc001000 - 0x000000fc009000 ACPI
0x00000100000000 - 0x000001fb200400 RAM

This is wrong because ACPI regions are also reported as RAM regions. The cause of this issue is not setting a big enough MMIO hole, current libxl code only takes into account the address of the local APIC page and the reserved pages in order to set the size of the MMIO hole, when it should also take into account the location of the ACPI tables.

After the fix the layout reported for the same guest is:

Quoted 8 lines
Quoted 6 lines

0x00000000000000 - 0x000000fc000000 RAM
0x000000fc000000 - 0x00000100000000 RESERVED
0x000000fc009000 - 0x000000fc009040 ACPI
0x000000fc000000 - 0x000000fc001000 ACPI
0x000000fc001000 - 0x000000fc009000 ACPI
0x00000100000000 - 0x000001fe000400 RAM

But this is still wrong - no two regions may overlap, regardless of type.

It's going to be more complicated to fix that. I can give it a try, but I think this is strictly better than what we do now.

Maybe instead of marking the whole MMIO hole as reserved we should only mark as reserved the lapic page and the special pages? That should avoid any overlaps.

Quoted 1 line

Also you appear to be losing all RAM in [fc009000, fee00000).

This is not populated. Note how:

0x000001fe000400 - 0x000001fb200400 == 0x000000fee00000 - 0x000000fc000000

The memory is shifted towards the end.

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#4 lore raw
Quoted 28 lines
Quoted 1 line
Quoted 1 line · On 20.04.18 at 16:15, <[email protected]> wrote:
Quoted 26 lines · Jan Beulich
Quoted 25 lines
Quoted 1 line
Quoted 1 line · On 20.04.18 at 15:52, <[email protected]> wrote:

PVH guests with 4GB of RAM or more get a memory map like the following:

0x00000000000000 - 0x000000fee00000 RAM
0x000000fee00000 - 0x00000100000000 RESERVED
0x000000fc009000 - 0x000000fc009040 ACPI
0x000000fc000000 - 0x000000fc001000 ACPI
0x000000fc001000 - 0x000000fc009000 ACPI
0x00000100000000 - 0x000001fb200400 RAM

This is wrong because ACPI regions are also reported as RAM regions. The cause of this issue is not setting a big enough MMIO hole, current libxl code only takes into account the address of the local APIC page and the reserved pages in order to set the size of the MMIO hole, when it should also take into account the location of the ACPI tables.

After the fix the layout reported for the same guest is:

0x00000000000000 - 0x000000fc000000 RAM
0x000000fc000000 - 0x00000100000000 RESERVED
0x000000fc009000 - 0x000000fc009040 ACPI
0x000000fc000000 - 0x000000fc001000 ACPI
0x000000fc001000 - 0x000000fc009000 ACPI
0x00000100000000 - 0x000001fe000400 RAM

Quoted 8 lines
Quoted 1 line

But this is still wrong - no two regions may overlap, regardless of type.

It's going to be more complicated to fix that. I can give it a try, but I think this is strictly better than what we do now.

Maybe instead of marking the whole MMIO hole as reserved we should only mark as reserved the lapic page and the special pages? That should avoid any overlaps.

Well, if nothing else is in that range (or can be placed there dynamically at runtime) I don't see why all of it is reserved. Marking a range reserved prevents, for example, PCI device BARs to be put there by the OS. The way it looks there's no MMIO window left available at all below 4Gb ...

Quoted 7 lines
Quoted 1 line

Also you appear to be losing all RAM in [fc009000, fee00000).

This is not populated. Note how:

0x000001fe000400 - 0x000001fb200400 == 0x000000fee00000 - 0x000000fc000000

The memory is shifted towards the end.

Oh, I see - I didn't pay attention to the upper bound of that last entry, I'm sorry.

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#5 lore raw
Quoted 33 lines · Jan Beulich
Quoted 33 lines
Quoted 1 line
Quoted 1 line · On 20.04.18 at 16:15, <[email protected]> wrote:
Quoted 19 lines · Jan Beulich
Quoted 19 lines
Quoted 1 line
Quoted 1 line · On 20.04.18 at 15:52, <[email protected]> wrote:

PVH guests with 4GB of RAM or more get a memory map like the following:

0x00000000000000 - 0x000000fee00000 RAM
0x000000fee00000 - 0x00000100000000 RESERVED
0x000000fc009000 - 0x000000fc009040 ACPI
0x000000fc000000 - 0x000000fc001000 ACPI
0x000000fc001000 - 0x000000fc009000 ACPI
0x00000100000000 - 0x000001fb200400 RAM

This is wrong because ACPI regions are also reported as RAM regions. The cause of this issue is not setting a big enough MMIO hole, current libxl code only takes into account the address of the local APIC page and the reserved pages in order to set the size of the MMIO hole, when it should also take into account the location of the ACPI tables.

After the fix the layout reported for the same guest is:

Quoted 8 lines
Quoted 6 lines

0x00000000000000 - 0x000000fc000000 RAM
0x000000fc000000 - 0x00000100000000 RESERVED
0x000000fc009000 - 0x000000fc009040 ACPI
0x000000fc000000 - 0x000000fc001000 ACPI
0x000000fc001000 - 0x000000fc009000 ACPI
0x00000100000000 - 0x000001fe000400 RAM

But this is still wrong - no two regions may overlap, regardless of type.

It's going to be more complicated to fix that. I can give it a try, but I think this is strictly better than what we do now.

Quoted 8 lines
Quoted 3 lines

Maybe instead of marking the whole MMIO hole as reserved we should only mark as reserved the lapic page and the special pages? That should avoid any overlaps.

Well, if nothing else is in that range (or can be placed there dynamically at runtime) I don't see why all of it is reserved. Marking a range reserved prevents, for example, PCI device BARs to be put there by the OS. The way it looks there's no MMIO window left available at all below 4Gb ...

Right, this will change when PVH gets devices with BARs, then it's going to need a proper MMIO hole below 4GB

AFAICT the only reserved pages for PVH are the HVM special pages, so I've changed the memory map builder to only report those as reserved, here is the resulting memory map:

0x00000000000000 - 0x000000fc000000 RAM
0x000000feff8000 - 0x000000ff000000 RESERVED
0x000000fc009000 - 0x000000fc009040 ACPI
0x000000fc000000 - 0x000000fc001000 ACPI
0x000000fc001000 - 0x000000fc009000 ACPI
0x00000100000000 - 0x000001fe000400 RAM

I see no reason to mark the local APIC page as reserved in the memory map. If that seems like a sensible approach I can send the updated patch.

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#6 lore raw
Quoted 11 lines
Quoted 1 line
Quoted 1 line · On 20.04.18 at 16:31, <[email protected]> wrote:

AFAICT the only reserved pages for PVH are the HVM special pages, so I've changed the memory map builder to only report those as reserved, here is the resulting memory map:

0x00000000000000 - 0x000000fc000000 RAM
0x000000feff8000 - 0x000000ff000000 RESERVED
0x000000fc009000 - 0x000000fc009040 ACPI
0x000000fc000000 - 0x000000fc001000 ACPI
0x000000fc001000 - 0x000000fc009000 ACPI
0x00000100000000 - 0x000001fe000400 RAM

LGTM

Quoted 3 lines

I see no reason to mark the local APIC page as reserved in the memory map. If that seems like a sensible approach I can send the updated patch.

Right - commonly bare hardware doesn't have this page marked reserved either.

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#7 lore raw
In reply to Roger Pau Monné
Quoted 35 lines · Roger Pau Monné
Quoted 34 lines · Jan Beulich
Quoted 34 lines
Quoted 1 line
Quoted 1 line · On 20.04.18 at 16:15, <[email protected]> wrote:
Quoted 19 lines · Jan Beulich
Quoted 19 lines
Quoted 1 line
Quoted 1 line · On 20.04.18 at 15:52, <[email protected]> wrote:

PVH guests with 4GB of RAM or more get a memory map like the following:

0x00000000000000 - 0x000000fee00000 RAM
0x000000fee00000 - 0x00000100000000 RESERVED
0x000000fc009000 - 0x000000fc009040 ACPI
0x000000fc000000 - 0x000000fc001000 ACPI
0x000000fc001000 - 0x000000fc009000 ACPI
0x00000100000000 - 0x000001fb200400 RAM

This is wrong because ACPI regions are also reported as RAM regions. The cause of this issue is not setting a big enough MMIO hole, current libxl code only takes into account the address of the local APIC page and the reserved pages in order to set the size of the MMIO hole, when it should also take into account the location of the ACPI tables.

After the fix the layout reported for the same guest is:

Quoted 7 lines
Quoted 6 lines

0x00000000000000 - 0x000000fc000000 RAM
0x000000fc000000 - 0x00000100000000 RESERVED
0x000000fc009000 - 0x000000fc009040 ACPI
0x000000fc000000 - 0x000000fc001000 ACPI
0x000000fc001000 - 0x000000fc009000 ACPI
0x00000100000000 - 0x000001fe000400 RAM

But this is still wrong - no two regions may overlap, regardless of type.

It's going to be more complicated to fix that. I can give it a try, but I think this is strictly better than what we do now.

Maybe instead of marking the whole MMIO hole as reserved we should only mark as reserved the lapic page and the special pages? That should avoid any overlaps.

Quoted 6 lines
Quoted 4 lines

Well, if nothing else is in that range (or can be placed there dynamically at runtime) I don't see why all of it is reserved. Marking a range reserved prevents, for example, PCI device BARs to be put there by the OS. The way it looks there's no MMIO window left available at all below 4Gb ...

Right, this will change when PVH gets devices with BARs, then it's going to need a proper MMIO hole below 4GB

Whomever wants to make PVH guests work properly fast:

The only way this work while retaining 1G host superpages is to unilaterally split at the 3G mark and continue from the 4G boundary. The ACPI tables and other ram-based tables can grow down from 3G, and there is plenty of room for the LAPIC to work, a sensibly sized MMIO hole, and gfn room to put mappings into without shattering superpages.

Then we've got to see about how to not use MTRRs...

~Andrew

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#8 lore raw
Quoted 43 lines · Andrew Cooper
Quoted 35 lines · Roger Pau Monné
Quoted 34 lines · Jan Beulich
Quoted 34 lines
Quoted 1 line
Quoted 1 line · On 20.04.18 at 16:15, <[email protected]> wrote:
Quoted 19 lines · Jan Beulich
Quoted 19 lines
Quoted 1 line
Quoted 1 line · On 20.04.18 at 15:52, <[email protected]> wrote:

PVH guests with 4GB of RAM or more get a memory map like the following:

0x00000000000000 - 0x000000fee00000 RAM
0x000000fee00000 - 0x00000100000000 RESERVED
0x000000fc009000 - 0x000000fc009040 ACPI
0x000000fc000000 - 0x000000fc001000 ACPI
0x000000fc001000 - 0x000000fc009000 ACPI
0x00000100000000 - 0x000001fb200400 RAM

This is wrong because ACPI regions are also reported as RAM regions. The cause of this issue is not setting a big enough MMIO hole, current libxl code only takes into account the address of the local APIC page and the reserved pages in order to set the size of the MMIO hole, when it should also take into account the location of the ACPI tables.

After the fix the layout reported for the same guest is:

Quoted 7 lines
Quoted 6 lines

0x00000000000000 - 0x000000fc000000 RAM
0x000000fc000000 - 0x00000100000000 RESERVED
0x000000fc009000 - 0x000000fc009040 ACPI
0x000000fc000000 - 0x000000fc001000 ACPI
0x000000fc001000 - 0x000000fc009000 ACPI
0x00000100000000 - 0x000001fe000400 RAM

But this is still wrong - no two regions may overlap, regardless of type.

It's going to be more complicated to fix that. I can give it a try, but I think this is strictly better than what we do now.

Maybe instead of marking the whole MMIO hole as reserved we should only mark as reserved the lapic page and the special pages? That should avoid any overlaps.

Quoted 6 lines
Quoted 4 lines

Well, if nothing else is in that range (or can be placed there dynamically at runtime) I don't see why all of it is reserved. Marking a range reserved prevents, for example, PCI device BARs to be put there by the OS. The way it looks there's no MMIO window left available at all below 4Gb ...

Right, this will change when PVH gets devices with BARs, then it's going to need a proper MMIO hole below 4GB

Quoted 7 lines

Whomever wants to make PVH guests work properly fast:

The only way this work while retaining 1G host superpages is to unilaterally split at the 3G mark and continue from the 4G boundary. The ACPI tables and other ram-based tables can grow down from 3G, and there is plenty of room for the LAPIC to work, a sensibly sized MMIO hole, and gfn room to put mappings into without shattering superpages.

Hmm, I'd rather put the ACPI tables between 3G and 4G. They are not accessed that often. The guest has then no need to use smaller pages for direct RAM mappings due to ACPI RO-mappings.

Juergen

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#9 lore raw
In reply to Andrew Cooper
Quoted 43 lines · Andrew Cooper
Quoted 35 lines · Roger Pau Monné
Quoted 34 lines · Jan Beulich
Quoted 34 lines
Quoted 1 line
Quoted 1 line · On 20.04.18 at 16:15, <[email protected]> wrote:
Quoted 19 lines · Jan Beulich
Quoted 19 lines
Quoted 1 line
Quoted 1 line · On 20.04.18 at 15:52, <[email protected]> wrote:

PVH guests with 4GB of RAM or more get a memory map like the following:

0x00000000000000 - 0x000000fee00000 RAM
0x000000fee00000 - 0x00000100000000 RESERVED
0x000000fc009000 - 0x000000fc009040 ACPI
0x000000fc000000 - 0x000000fc001000 ACPI
0x000000fc001000 - 0x000000fc009000 ACPI
0x00000100000000 - 0x000001fb200400 RAM

This is wrong because ACPI regions are also reported as RAM regions. The cause of this issue is not setting a big enough MMIO hole, current libxl code only takes into account the address of the local APIC page and the reserved pages in order to set the size of the MMIO hole, when it should also take into account the location of the ACPI tables.

After the fix the layout reported for the same guest is:

Quoted 7 lines
Quoted 6 lines

0x00000000000000 - 0x000000fc000000 RAM
0x000000fc000000 - 0x00000100000000 RESERVED
0x000000fc009000 - 0x000000fc009040 ACPI
0x000000fc000000 - 0x000000fc001000 ACPI
0x000000fc001000 - 0x000000fc009000 ACPI
0x00000100000000 - 0x000001fe000400 RAM

But this is still wrong - no two regions may overlap, regardless of type.

It's going to be more complicated to fix that. I can give it a try, but I think this is strictly better than what we do now.

Maybe instead of marking the whole MMIO hole as reserved we should only mark as reserved the lapic page and the special pages? That should avoid any overlaps.

Quoted 6 lines
Quoted 4 lines

Well, if nothing else is in that range (or can be placed there dynamically at runtime) I don't see why all of it is reserved. Marking a range reserved prevents, for example, PCI device BARs to be put there by the OS. The way it looks there's no MMIO window left available at all below 4Gb ...

Right, this will change when PVH gets devices with BARs, then it's going to need a proper MMIO hole below 4GB

Quoted 7 lines

Whomever wants to make PVH guests work properly fast:

The only way this work while retaining 1G host superpages is to unilaterally split at the 3G mark and continue from the 4G boundary. The ACPI tables and other ram-based tables can grow down from 3G, and there is plenty of room for the LAPIC to work, a sensibly sized MMIO hole, and gfn room to put mappings into without shattering superpages.

Just having the hole from 3G to 4G and the ACPI tables somewhere in there seems good enough. At the end of day ACPI tables are not accessed that often.

In any case, that's 4.12 material.

Roger.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel