libxl: fix memory map reported to PVH guests
PATCH 1/1 libxl: fix memory map reported to PVH guests
PVH guests with 4GB of RAM or more get a memory map like the following:
0x00000000000000 - 0x000000fee00000 RAM0x000000fee00000 - 0x00000100000000 RESERVED0x000000fc009000 - 0x000000fc009040 ACPI0x000000fc000000 - 0x000000fc001000 ACPI0x000000fc001000 - 0x000000fc009000 ACPI0x00000100000000 - 0x000001fb200400 RAM
This is wrong because ACPI regions are also reported as RAM regions. The cause of this issue is not setting a big enough MMIO hole, current libxl code only takes into account the address of the local APIC page and the reserved pages in order to set the size of the MMIO hole, when it should also take into account the location of the ACPI tables.
After the fix the layout reported for the same guest is:
0x00000000000000 - 0x000000fc000000 RAM0x000000fc000000 - 0x00000100000000 RESERVED0x000000fc009000 - 0x000000fc009040 ACPI0x000000fc000000 - 0x000000fc001000 ACPI0x000000fc001000 - 0x000000fc009000 ACPI0x00000100000000 - 0x000001fe000400 RAM
---
Cc: Ian Jackson <[email protected]>
Cc: Wei Liu <[email protected]>
Cc: Juergen Gross <[email protected]>
--- tools/libxl/libxl_arch.h | 1 + tools/libxl/libxl_dom.c | 20 +++++++++++--------- tools/libxl/libxl_x86_acpi.c | 1 - 3 files changed, 12 insertions(+), 10 deletions(-)
tools/libxl/libxl_arch.h +1 -0
@@ -76,6 +76,7 @@ int libxl__arch_extra_memory(libxl__gc *gc,#if defined(__i386__) || defined(__x86_64__)#define LAPIC_BASE_ADDRESS 0xfee00000 +#define ACPI_INFO_PHYSICAL_ADDRESS 0xfc000000 int libxl__dom_load_acpi(libxl__gc *gc, const libxl_domain_build_info *b_info,
tools/libxl/libxl_dom.c +11 -9
@@ -1229,15 +1229,17 @@ int libxl__build_hvm(libxl__gc *gc, uint32_t domid, dom->mmio_size =HVM_BELOW_4G_MMIO_LENGTH ; else if (dom->mmio_size == 0 && !device_model) {#if defined(__i386__) || defined(__x86_64__) -if (libxl_defbool_val (info->apic)) { -/* Make sure LAPIC_BASE_ADDRESS is below special pages */ -assert (((((X86_HVM_END_SPECIAL_REGION -X86_HVM_NR_SPECIAL_PAGES ) - <<XC_PAGE_SHIFT ) -LAPIC_BASE_ADDRESS )) >=XC_PAGE_SIZE ); - dom->mmio_size =GB (4 ) -LAPIC_BASE_ADDRESS ; - }else - dom->mmio_size =GB (4 ) - - ((X86_HVM_END_SPECIAL_REGION -X86_HVM_NR_SPECIAL_PAGES ) - <<XC_PAGE_SHIFT ); +/* +* Make sure the local APIC page, the ACPI tables and the special pages +* are inside the MMIO hole. +*/ +xen_paddr_t start = + (X86_HVM_END_SPECIAL_REGION -X86_HVM_NR_SPECIAL_PAGES ) << +XC_PAGE_SHIFT ; + + start =min_t (xen_paddr_t, start,LAPIC_BASE_ADDRESS ); + start =min_t (xen_paddr_t, start,ACPI_INFO_PHYSICAL_ADDRESS ); + dom->mmio_size =GB (4 ) - start;#else assert (1 );#endif
tools/libxl/libxl_x86_acpi.c +0 -1
@@ -22,7 +22,6 @@/* Number of pages holding ACPI tables */ #define NUM_ACPI_PAGES 16 -#define ACPI_INFO_PHYSICAL_ADDRESS 0xfc000000struct libxl_acpi_ctxt {struct acpi_ctxt c;
Signature · 7 lines
2.17.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 18 lines · Roger Pau Monne
PVH guests with 4GB of RAM or more get a memory map like the following:
0x00000000000000-0x000000fee00000RAM0x000000fee00000-0x00000100000000RESERVED0x000000fc009000-0x000000fc009040ACPI0x000000fc000000-0x000000fc001000ACPI0x000000fc001000-0x000000fc009000ACPI0x00000100000000-0x000001fb200400RAMThis is wrong because ACPI regions are also reported as RAM regions. The cause of this issue is not setting a big enough MMIO hole, current libxl code only takes into account the address of the local APIC page and the reserved pages in order to set the size of the MMIO hole, when it should also take into account the location of the ACPI tables.
After the fix the layout reported for the same guest is:
Quoted 8 lines
0x00000000000000-0x000000fc000000RAM0x000000fc000000-0x00000100000000RESERVED0x000000fc009000-0x000000fc009040ACPI0x000000fc000000-0x000000fc001000ACPI0x000000fc001000-0x000000fc009000ACPI0x00000100000000-0x000001fe000400RAMSigned-off-by Roger Pau Monné <[email protected]>
Juergen
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 17 lines
Quoted 1 line
Quoted 1 line · On 20.04.18 at 15:52, <[email protected]> wrote:
PVH guests with 4GB of RAM or more get a memory map like the following:
0x00000000000000-0x000000fee00000RAM0x000000fee00000-0x00000100000000RESERVED0x000000fc009000-0x000000fc009040ACPI0x000000fc000000-0x000000fc001000ACPI0x000000fc001000-0x000000fc009000ACPI0x00000100000000-0x000001fb200400RAMThis is wrong because ACPI regions are also reported as RAM regions. The cause of this issue is not setting a big enough MMIO hole, current libxl code only takes into account the address of the local APIC page and the reserved pages in order to set the size of the MMIO hole, when it should also take into account the location of the ACPI tables.
Quoted 8 lines
After the fix the layout reported for the same guest is:
0x00000000000000-0x000000fc000000RAM0x000000fc000000-0x00000100000000RESERVED0x000000fc009000-0x000000fc009040ACPI0x000000fc000000-0x000000fc001000ACPI0x000000fc001000-0x000000fc009000ACPI0x00000100000000-0x000001fe000400RAM
But this is still wrong - no two regions may overlap, regardless of type. Also you appear to be losing all RAM in [fc009000, fee00000).
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 19 lines · Jan Beulich
Quoted 19 lines
Quoted 1 line
Quoted 1 line · On 20.04.18 at 15:52, <[email protected]> wrote:
PVH guests with 4GB of RAM or more get a memory map like the following:
0x00000000000000-0x000000fee00000RAM0x000000fee00000-0x00000100000000RESERVED0x000000fc009000-0x000000fc009040ACPI0x000000fc000000-0x000000fc001000ACPI0x000000fc001000-0x000000fc009000ACPI0x00000100000000-0x000001fb200400RAMThis is wrong because ACPI regions are also reported as RAM regions. The cause of this issue is not setting a big enough MMIO hole, current libxl code only takes into account the address of the local APIC page and the reserved pages in order to set the size of the MMIO hole, when it should also take into account the location of the ACPI tables.
After the fix the layout reported for the same guest is:
Quoted 8 lines
Quoted 6 lines
0x00000000000000-0x000000fc000000RAM0x000000fc000000-0x00000100000000RESERVED0x000000fc009000-0x000000fc009040ACPI0x000000fc000000-0x000000fc001000ACPI0x000000fc001000-0x000000fc009000ACPI0x00000100000000-0x000001fe000400RAMBut this is still wrong - no two regions may overlap, regardless of type.
It's going to be more complicated to fix that. I can give it a try, but I think this is strictly better than what we do now.
Maybe instead of marking the whole MMIO hole as reserved we should only mark as reserved the lapic page and the special pages? That should avoid any overlaps.
Quoted 1 line
Also you appear to be losing all RAM in [
fc009000,fee00000).
This is not populated. Note how:
0x000001fe000400 - 0x000001fb200400 == 0x000000fee00000 - 0x000000fc000000
The memory is shifted towards the end.
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 28 lines
Quoted 1 line
Quoted 1 line · On 20.04.18 at 16:15, <[email protected]> wrote:
Quoted 26 lines · Jan Beulich
Quoted 25 lines
Quoted 1 line
Quoted 1 line · On 20.04.18 at 15:52, <[email protected]> wrote:
PVH guests with 4GB of RAM or more get a memory map like the following:
0x00000000000000-0x000000fee00000RAM0x000000fee00000-0x00000100000000RESERVED0x000000fc009000-0x000000fc009040ACPI0x000000fc000000-0x000000fc001000ACPI0x000000fc001000-0x000000fc009000ACPI0x00000100000000-0x000001fb200400RAMThis is wrong because ACPI regions are also reported as RAM regions. The cause of this issue is not setting a big enough MMIO hole, current libxl code only takes into account the address of the local APIC page and the reserved pages in order to set the size of the MMIO hole, when it should also take into account the location of the ACPI tables.
After the fix the layout reported for the same guest is:
0x00000000000000-0x000000fc000000RAM0x000000fc000000-0x00000100000000RESERVED0x000000fc009000-0x000000fc009040ACPI0x000000fc000000-0x000000fc001000ACPI0x000000fc001000-0x000000fc009000ACPI0x00000100000000-0x000001fe000400RAM
Quoted 8 lines
Quoted 1 line
But this is still wrong - no two regions may overlap, regardless of type.
It's going to be more complicated to fix that. I can give it a try, but I think this is strictly better than what we do now.
Maybe instead of marking the whole MMIO hole as reserved we should only mark as reserved the lapic page and the special pages? That should avoid any overlaps.
Well, if nothing else is in that range (or can be placed there dynamically at runtime) I don't see why all of it is reserved. Marking a range reserved prevents, for example, PCI device BARs to be put there by the OS. The way it looks there's no MMIO window left available at all below 4Gb ...
Quoted 7 lines
Quoted 1 line
Also you appear to be losing all RAM in [
fc009000,fee00000).This is not populated. Note how:
0x000001fe000400-0x000001fb200400==0x000000fee00000-0x000000fc000000The memory is shifted towards the end.
Oh, I see - I didn't pay attention to the upper bound of that last entry, I'm sorry.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 33 lines · Jan Beulich
Quoted 33 lines
Quoted 1 line
Quoted 1 line · On 20.04.18 at 16:15, <[email protected]> wrote:
Quoted 19 lines · Jan Beulich
Quoted 19 lines
Quoted 1 line
Quoted 1 line · On 20.04.18 at 15:52, <[email protected]> wrote:
PVH guests with 4GB of RAM or more get a memory map like the following:
0x00000000000000-0x000000fee00000RAM0x000000fee00000-0x00000100000000RESERVED0x000000fc009000-0x000000fc009040ACPI0x000000fc000000-0x000000fc001000ACPI0x000000fc001000-0x000000fc009000ACPI0x00000100000000-0x000001fb200400RAMThis is wrong because ACPI regions are also reported as RAM regions. The cause of this issue is not setting a big enough MMIO hole, current libxl code only takes into account the address of the local APIC page and the reserved pages in order to set the size of the MMIO hole, when it should also take into account the location of the ACPI tables.
After the fix the layout reported for the same guest is:
Quoted 8 lines
Quoted 6 lines
0x00000000000000-0x000000fc000000RAM0x000000fc000000-0x00000100000000RESERVED0x000000fc009000-0x000000fc009040ACPI0x000000fc000000-0x000000fc001000ACPI0x000000fc001000-0x000000fc009000ACPI0x00000100000000-0x000001fe000400RAMBut this is still wrong - no two regions may overlap, regardless of type.
It's going to be more complicated to fix that. I can give it a try, but I think this is strictly better than what we do now.
Quoted 8 lines
Quoted 3 lines
Maybe instead of marking the whole MMIO hole as reserved we should only mark as reserved the lapic page and the special pages? That should avoid any overlaps.
Well, if nothing else is in that range (or can be placed there dynamically at runtime) I don't see why all of it is reserved. Marking a range reserved prevents, for example, PCI device BARs to be put there by the OS. The way it looks there's no MMIO window left available at all below 4Gb ...
Right, this will change when PVH gets devices with BARs, then it's going to need a proper MMIO hole below 4GB
AFAICT the only reserved pages for PVH are the HVM special pages, so I've changed the memory map builder to only report those as reserved, here is the resulting memory map:
0x00000000000000 - 0x000000fc000000 RAM0x000000feff8000 - 0x000000ff000000 RESERVED0x000000fc009000 - 0x000000fc009040 ACPI0x000000fc000000 - 0x000000fc001000 ACPI0x000000fc001000 - 0x000000fc009000 ACPI0x00000100000000 - 0x000001fe000400 RAM
I see no reason to mark the local APIC page as reserved in the memory map. If that seems like a sensible approach I can send the updated patch.
Thanks, Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 11 lines
Quoted 1 line
Quoted 1 line · On 20.04.18 at 16:31, <[email protected]> wrote:
AFAICT the only reserved pages for PVH are the HVM special pages, so I've changed the memory map builder to only report those as reserved, here is the resulting memory map:
0x00000000000000-0x000000fc000000RAM0x000000feff8000-0x000000ff000000RESERVED0x000000fc009000-0x000000fc009040ACPI0x000000fc000000-0x000000fc001000ACPI0x000000fc001000-0x000000fc009000ACPI0x00000100000000-0x000001fe000400RAM
LGTM
Quoted 3 lines
I see no reason to mark the local APIC page as reserved in the memory map. If that seems like a sensible approach I can send the updated patch.
Right - commonly bare hardware doesn't have this page marked reserved either.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 35 lines · Roger Pau Monné
Quoted 34 lines · Jan Beulich
Quoted 34 lines
Quoted 1 line
Quoted 1 line · On 20.04.18 at 16:15, <[email protected]> wrote:
Quoted 19 lines · Jan Beulich
Quoted 19 lines
Quoted 1 line
Quoted 1 line · On 20.04.18 at 15:52, <[email protected]> wrote:
PVH guests with 4GB of RAM or more get a memory map like the following:
0x00000000000000-0x000000fee00000RAM0x000000fee00000-0x00000100000000RESERVED0x000000fc009000-0x000000fc009040ACPI0x000000fc000000-0x000000fc001000ACPI0x000000fc001000-0x000000fc009000ACPI0x00000100000000-0x000001fb200400RAMThis is wrong because ACPI regions are also reported as RAM regions. The cause of this issue is not setting a big enough MMIO hole, current libxl code only takes into account the address of the local APIC page and the reserved pages in order to set the size of the MMIO hole, when it should also take into account the location of the ACPI tables.
After the fix the layout reported for the same guest is:
Quoted 7 lines
Quoted 6 lines
0x00000000000000-0x000000fc000000RAM0x000000fc000000-0x00000100000000RESERVED0x000000fc009000-0x000000fc009040ACPI0x000000fc000000-0x000000fc001000ACPI0x000000fc001000-0x000000fc009000ACPI0x00000100000000-0x000001fe000400RAMBut this is still wrong - no two regions may overlap, regardless of type.
It's going to be more complicated to fix that. I can give it a try, but I think this is strictly better than what we do now.
Maybe instead of marking the whole MMIO hole as reserved we should only mark as reserved the lapic page and the special pages? That should avoid any overlaps.
Quoted 6 lines
Quoted 4 lines
Well, if nothing else is in that range (or can be placed there dynamically at runtime) I don't see why all of it is reserved. Marking a range reserved prevents, for example, PCI device BARs to be put there by the OS. The way it looks there's no MMIO window left available at all below 4Gb ...
Right, this will change when PVH gets devices with BARs, then it's going to need a proper MMIO hole below 4GB
Whomever wants to make PVH guests work properly fast:
The only way this work while retaining 1G host superpages is to unilaterally split at the 3G mark and continue from the 4G boundary. The ACPI tables and other ram-based tables can grow down from 3G, and there is plenty of room for the LAPIC to work, a sensibly sized MMIO hole, and gfn room to put mappings into without shattering superpages.
Then we've got to see about how to not use MTRRs...
~Andrew
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 43 lines · Andrew Cooper
Quoted 35 lines · Roger Pau Monné
Quoted 34 lines · Jan Beulich
Quoted 34 lines
Quoted 1 line
Quoted 1 line · On 20.04.18 at 16:15, <[email protected]> wrote:
Quoted 19 lines · Jan Beulich
Quoted 19 lines
Quoted 1 line
Quoted 1 line · On 20.04.18 at 15:52, <[email protected]> wrote:
PVH guests with 4GB of RAM or more get a memory map like the following:
0x00000000000000-0x000000fee00000RAM0x000000fee00000-0x00000100000000RESERVED0x000000fc009000-0x000000fc009040ACPI0x000000fc000000-0x000000fc001000ACPI0x000000fc001000-0x000000fc009000ACPI0x00000100000000-0x000001fb200400RAMThis is wrong because ACPI regions are also reported as RAM regions. The cause of this issue is not setting a big enough MMIO hole, current libxl code only takes into account the address of the local APIC page and the reserved pages in order to set the size of the MMIO hole, when it should also take into account the location of the ACPI tables.
After the fix the layout reported for the same guest is:
Quoted 7 lines
Quoted 6 lines
0x00000000000000-0x000000fc000000RAM0x000000fc000000-0x00000100000000RESERVED0x000000fc009000-0x000000fc009040ACPI0x000000fc000000-0x000000fc001000ACPI0x000000fc001000-0x000000fc009000ACPI0x00000100000000-0x000001fe000400RAMBut this is still wrong - no two regions may overlap, regardless of type.
It's going to be more complicated to fix that. I can give it a try, but I think this is strictly better than what we do now.
Maybe instead of marking the whole MMIO hole as reserved we should only mark as reserved the lapic page and the special pages? That should avoid any overlaps.
Quoted 6 lines
Quoted 4 lines
Well, if nothing else is in that range (or can be placed there dynamically at runtime) I don't see why all of it is reserved. Marking a range reserved prevents, for example, PCI device BARs to be put there by the OS. The way it looks there's no MMIO window left available at all below 4Gb ...
Right, this will change when PVH gets devices with BARs, then it's going to need a proper MMIO hole below 4GB
Quoted 7 lines
Whomever wants to make PVH guests work properly fast:
The only way this work while retaining 1G host superpages is to unilaterally split at the 3G mark and continue from the 4G boundary. The ACPI tables and other ram-based tables can grow down from 3G, and there is plenty of room for the LAPIC to work, a sensibly sized MMIO hole, and gfn room to put mappings into without shattering superpages.
Hmm, I'd rather put the ACPI tables between 3G and 4G. They are not accessed that often. The guest has then no need to use smaller pages for direct RAM mappings due to ACPI RO-mappings.
Juergen
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 43 lines · Andrew Cooper
Quoted 35 lines · Roger Pau Monné
Quoted 34 lines · Jan Beulich
Quoted 34 lines
Quoted 1 line
Quoted 1 line · On 20.04.18 at 16:15, <[email protected]> wrote:
Quoted 19 lines · Jan Beulich
Quoted 19 lines
Quoted 1 line
Quoted 1 line · On 20.04.18 at 15:52, <[email protected]> wrote:
PVH guests with 4GB of RAM or more get a memory map like the following:
0x00000000000000-0x000000fee00000RAM0x000000fee00000-0x00000100000000RESERVED0x000000fc009000-0x000000fc009040ACPI0x000000fc000000-0x000000fc001000ACPI0x000000fc001000-0x000000fc009000ACPI0x00000100000000-0x000001fb200400RAMThis is wrong because ACPI regions are also reported as RAM regions. The cause of this issue is not setting a big enough MMIO hole, current libxl code only takes into account the address of the local APIC page and the reserved pages in order to set the size of the MMIO hole, when it should also take into account the location of the ACPI tables.
After the fix the layout reported for the same guest is:
Quoted 7 lines
Quoted 6 lines
0x00000000000000-0x000000fc000000RAM0x000000fc000000-0x00000100000000RESERVED0x000000fc009000-0x000000fc009040ACPI0x000000fc000000-0x000000fc001000ACPI0x000000fc001000-0x000000fc009000ACPI0x00000100000000-0x000001fe000400RAMBut this is still wrong - no two regions may overlap, regardless of type.
It's going to be more complicated to fix that. I can give it a try, but I think this is strictly better than what we do now.
Maybe instead of marking the whole MMIO hole as reserved we should only mark as reserved the lapic page and the special pages? That should avoid any overlaps.
Quoted 6 lines
Quoted 4 lines
Well, if nothing else is in that range (or can be placed there dynamically at runtime) I don't see why all of it is reserved. Marking a range reserved prevents, for example, PCI device BARs to be put there by the OS. The way it looks there's no MMIO window left available at all below 4Gb ...
Right, this will change when PVH gets devices with BARs, then it's going to need a proper MMIO hole below 4GB
Quoted 7 lines
Whomever wants to make PVH guests work properly fast:
The only way this work while retaining 1G host superpages is to unilaterally split at the 3G mark and continue from the 4G boundary. The ACPI tables and other ram-based tables can grow down from 3G, and there is plenty of room for the LAPIC to work, a sensibly sized MMIO hole, and gfn room to put mappings into without shattering superpages.
Just having the hole from 3G to 4G and the ACPI tables somewhere in there seems good enough. At the end of day ACPI tables are not accessed that often.
In any case, that's 4.12 material.
Roger.
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel