Cannot boot PVH dom0 with big initrd
Hi,
After fixing the xhci crash, I hit another issue - booting with 236MB initrd doesn't work, I get:
(XEN) [ 3.151856] *** Building a PVH Dom0 ***
...
(XEN) [ 3.593940] Unable to allocate memory with order 0!
(XEN) [ 3.597110] Failed to setup Dom0 physical memory map
(XEN) [ 3.599884]
(XEN) [ 3.602482] ****************************************
(XEN) [ 3.605272] Panic on CPU 0:
(XEN) [ 3.607928] Could not construct d0
(XEN) [ 3.610692] ****************************************
(XEN) [ 3.613463]
(XEN) [ 3.616035] Reboot in five seconds...
(XEN) [ 8.626565] Resetting with ACPI MEMORY or I/O RESET_REG.Full console log: https://gist.github.com/marmarek/c9dbc87bf07b76f2899781755762f565
If I skip initrd, then it boots just fine (but dom0 is not happy about that). 164MB initrd failed too, but 13MB started ok.
Just in case, I tried skipping XHCI console, but it didn't change anything.
Host has 16GB of memory, and there is no dom0_mem= parameter. Xen is started from GRUB, using MB2+EFI.
When it works (12MB initrd), I get the following info:
(XEN) [ 4.123843] Dom0 memory allocation stats:
(XEN) [ 4.126636] order 0 allocations: 4
(XEN) [ 4.129278] order 1 allocations: 3
(XEN) [ 4.132027] order 2 allocations: 4
(XEN) [ 4.134776] order 3 allocations: 3
(XEN) [ 4.137527] order 4 allocations: 3
(XEN) [ 4.140252] order 5 allocations: 3
(XEN) [ 4.142966] order 7 allocations: 2
(XEN) [ 4.145663] order 8 allocations: 3
(XEN) [ 4.148358] order 9 allocations: 3
(XEN) [ 4.151035] order 10 allocations: 4
(XEN) [ 4.153709] order 11 allocations: 7
(XEN) [ 4.156277] order 12 allocations: 9
(XEN) [ 4.158940] order 13 allocations: 6
(XEN) [ 4.161604] order 14 allocations: 6
(XEN) [ 4.164251] order 15 allocations: 7
(XEN) [ 4.166892] order 16 allocations: 6
(XEN) [ 4.169521] order 17 allocations: 4
(XEN) [ 4.172048] order 18 allocations: 10
(XEN) [ 4.994309] ELF: phdr: paddr=0x200000 memsz=0x1ff3928
(XEN) [ 4.997011] ELF: phdr: paddr=0x2200000 memsz=0x1c00000
(XEN) [ 4.999686] ELF: memory: 0x200000 -> 0x3e00000
(XEN) [ 5.002404] ELF: note: PHYS32_RELOC align: 0x200000 min: 0x200000 max: 0x3fffffff
(XEN) [ 5.005451] ELF: note: PHYS32_ENTRY = 0x16a2ca0
(XEN) [ 5.008519] ELF: note: GUEST_OS = "linux"
(XEN) [ 5.011562] ELF: note: GUEST_VERSION = "2.6"
(XEN) [ 5.014634] ELF: note: XEN_VERSION = "xen-3.0"
(XEN) [ 5.017712] ELF: note: VIRT_BASE = 0xffffffff80000000
(XEN) [ 5.020795] ELF: note: INIT_P2M = 0x8000000000
(XEN) [ 5.023856] ELF: note: ENTRY = 0xffffffff82d3c160
(XEN) [ 5.026924] ELF: note: FEATURES = "!writable_page_tables"
(XEN) [ 5.029976] ELF: note: PAE_MODE = "yes"
(XEN) [ 5.032882] ELF: note: L1_MFN_VALID
(XEN) [ 5.035516] ELF: note: MOD_START_PFN = 0x1
(XEN) [ 5.038442] ELF: note: PADDR_OFFSET = 0
(XEN) [ 5.041250] ELF: note: SUPPORTED_FEATURES = 0x8801
(XEN) [ 5.044169] ELF: note: LOADER = "generic"
(XEN) [ 5.047048] ELF: note: SUSPEND_CANCEL = 0x1
(XEN) [ 5.049931] ELF: Found PVH image
(XEN) [ 5.052712] ELF: addresses:
(XEN) [ 5.055090] virt_base = 0x0
(XEN) [ 5.057552] elf_paddr_offset = 0x0
(XEN) [ 5.060007] virt_offset = 0x0
(XEN) [ 5.062476] virt_kstart = 0x200000
(XEN) [ 5.064924] virt_kend = 0x3e00000
(XEN) [ 5.067380] virt_entry = 0x16a2ca0
(XEN) [ 5.069841] p2m_base = 0x8000000000
(XEN) [ 5.072319] ELF: phdr 0 at 0x200000 -> 0x21f3928
(XEN) [ 5.080076] ELF: phdr 1 at 0x2200000 -> 0x3e00000
(XEN) [ 5.090182] Dom0 memory map:
(XEN) [ 5.092531] [0000000000000000, 000000000009efff] (usable)
(XEN) [ 5.095086] [000000000009f000, 00000000000fffff] (reserved)
(XEN) [ 5.097625] [0000000000100000, 000000005471afff] (usable)
(XEN) [ 5.100156] [000000005471b000, 000000005475bfff] (reserved)
(XEN) [ 5.102704] [000000005475c000, 0000000063c2dfff] (usable)
(XEN) [ 5.105259] [0000000063c2e000, 000000006d17afff] (reserved)
(XEN) [ 5.107853] [000000006d17b000, 000000006d22bfff] (ACPI data)
(XEN) [ 5.110459] [000000006d22c000, 000000006d2ebfff] (ACPI NVS)
(XEN) [ 5.113082] [000000006d2ec000, 000000006fffefff] (reserved)
(XEN) [ 5.115726] [000000006ffff000, 000000006ffffdcb] (usable)
(XEN) [ 5.118388] [000000006ffffdcc, 000000006ffffe97] (ACPI data)
(XEN) [ 5.121080] [0000000070000000, 00000000807fffff] (reserved)
(XEN) [ 5.123776] [00000000c0000000, 00000000cfffffff] (reserved)
(XEN) [ 5.126498] [00000000fe000000, 00000000fe010fff] (reserved)
(XEN) [ 5.129244] [00000000fec00000, 00000000fec00fff] (reserved)
(XEN) [ 5.132001] [00000000fed00000, 00000000fed00fff] (reserved)
(XEN) [ 5.134797] [00000000fed20000, 00000000fed7ffff] (reserved)
(XEN) [ 5.137619] [00000000fee00000, 00000000fee00fff] (reserved)
(XEN) [ 5.140436] [00000000ff000000, 00000001023fffff] (reserved)
(XEN) [ 5.143271] [0000000102400000, 0000000468b34fff] (usable)
(XEN) [ 5.146131] [0000000468b35000, 000000047f7fffff] (unusable)
(XEN) [ 5.149015] Initial low memory virq threshold set at 0x4000 pages.
(XEN) [ 5.151852] Scrubbing Free RAM in background
(XEN) [ 5.154667] Std. Loglevel: All
(XEN) [ 5.157465] Guest Loglevel: All
(XEN) [ 5.160250] Xen is relinquishing VGA console.
(XEN) [ 5.166762] *** Serial input to DOM0 (type 'CTRL-a' three times to switch input)
(XEN) [ 5.167550] Re-running stub recovery selftests...
(XEN) [ 5.167692] Fixup #UD[0000]: ffff82d07fffe044 [ffff82d07fffe044] -> ffff82d0403b8753
(XEN) [ 5.167914] Fixup #GP[0000]: ffff82d07fffe045 [ffff82d07fffe045] -> ffff82d0403b8753
(XEN) [ 5.168139] Fixup #SS[0000]: ffff82d07fffe044 [ffff82d07fffe044] -> ffff82d0403b8753
(XEN) [ 5.168359] Fixup #BP[0000]: ffff82d07fffe045 [ffff82d07fffe045] -> ffff82d0403b8753
(XEN) [ 5.168662] Freed 720kB init memory
(XEN) [ 6.884758] d0v0: upcall vector f3Interestingly, this appear to have worked on Xen 4.19.3.
Signature · 3 lines
Best Regards, Marek Marczykowski-Górecki Invisible Things Lab
Quoted 20 lines · Marek Marczykowski-Górecki
Hi,
After fixing the xhci crash, I hit another issue - booting with 236MB initrd doesn't work, I get:
(XEN) [ 3.151856] *** Building a PVH Dom0 *** ... (XEN) [ 3.593940] Unable to allocate memory with order 0! (XEN) [ 3.597110] Failed to setup Dom0 physical memory map (XEN) [ 3.599884] (XEN) [ 3.602482] **************************************** (XEN) [ 3.605272] Panic on CPU 0: (XEN) [ 3.607928] Could not construct d0 (XEN) [ 3.610692] **************************************** (XEN) [ 3.613463] (XEN) [ 3.616035] Reboot in five seconds... (XEN) [ 8.626565] Resetting with ACPI MEMORY or I/O RESET_REG.Full console log: https://gist.github.com/marmarek/c9dbc87bf07b76f2899781755762f565
Quoted 7 lines
If I skip initrd, then it boots just fine (but dom0 is not happy about that). 164MB initrd failed too, but 13MB started ok.
Just in case, I tried skipping XHCI console, but it didn't change anything.Host has 16GB of memory, and there is no dom0_mem= parameter. Xen is started from GRUB, using MB2+EFI.
Hmm, yes, there's an ordering issue: Of course we free initrd space (as used for passing from the boot loader to Xen) only after copying to the designated guest area. Yet dom0_compute_nr_pages(), intentionally, includes the space in its calculation (adding initial_images_nrpages()'s return value). PV Dom0 isn't affected because to load huge initrd there, the kernel has to request the initrd to not be mapped into the initial allocation.
Jan
Quoted 29 lines · Jan Beulich
Quoted 20 lines · Marek Marczykowski-Górecki
Hi,
After fixing the xhci crash, I hit another issue - booting with 236MB initrd doesn't work, I get:
(XEN) [ 3.151856] *** Building a PVH Dom0 *** ... (XEN) [ 3.593940] Unable to allocate memory with order 0! (XEN) [ 3.597110] Failed to setup Dom0 physical memory map (XEN) [ 3.599884] (XEN) [ 3.602482] **************************************** (XEN) [ 3.605272] Panic on CPU 0: (XEN) [ 3.607928] Could not construct d0 (XEN) [ 3.610692] **************************************** (XEN) [ 3.613463] (XEN) [ 3.616035] Reboot in five seconds... (XEN) [ 8.626565] Resetting with ACPI MEMORY or I/O RESET_REG.Full console log: https://gist.github.com/marmarek/c9dbc87bf07b76f2899781755762f565
Quoted 7 lines
If I skip initrd, then it boots just fine (but dom0 is not happy about that). 164MB initrd failed too, but 13MB started ok.
Just in case, I tried skipping XHCI console, but it didn't change anything.Host has 16GB of memory, and there is no dom0_mem= parameter. Xen is started from GRUB, using MB2+EFI.
Quoted 6 lines
Hmm, yes, there's an ordering issue: Of course we free initrd space (as used for passing from the boot loader to Xen) only after copying to the designated guest area. Yet
dom0_compute_nr_pages(), intentionally, includes the space in its calculation (adding initial_images_nrpages()'s return value). PV Dom0 isn't affected because to load huge initrd there, the kernel has to request the initrd to not be mapped into the initial allocation.
Right, on PV dom0 we do not copy the image to a new set of pages, we simply assign the pages where the initrd resides to the domain. We can't populate those pages in the p2m as-is, otherwise we would shatter super pages.
I think the fix below should do it, it's likely the best we can do. Can you please give it a try Marek?
Thanks, Roger.
---
xen/arch/x86/dom0_build.c +1 -1
@@ -343,7 +343,7 @@ unsigned long __init dom0_compute_nr_pages(for_each_node_mask ( node, dom0_nodes ) avail += avail_domheap_pages_region(node, 0, 0) + -initial_images_nrpages (node); +is_pv_domain (d) ?initial_images_nrpages (node) :0 ;/* Reserve memory for further dom0 vcpu-struct allocations... */ avail -= (d->max_vcpus -1UL )
Quoted 52 lines · Roger Pau Monné
Quoted 29 lines · Jan Beulich
Quoted 20 lines · Marek Marczykowski-Górecki
Hi,
After fixing the xhci crash, I hit another issue - booting with 236MB initrd doesn't work, I get:
(XEN) [ 3.151856] *** Building a PVH Dom0 *** ... (XEN) [ 3.593940] Unable to allocate memory with order 0! (XEN) [ 3.597110] Failed to setup Dom0 physical memory map (XEN) [ 3.599884] (XEN) [ 3.602482] **************************************** (XEN) [ 3.605272] Panic on CPU 0: (XEN) [ 3.607928] Could not construct d0 (XEN) [ 3.610692] **************************************** (XEN) [ 3.613463] (XEN) [ 3.616035] Reboot in five seconds... (XEN) [ 8.626565] Resetting with ACPI MEMORY or I/O RESET_REG.Full console log: https://gist.github.com/marmarek/c9dbc87bf07b76f2899781755762f565
Quoted 7 lines
If I skip initrd, then it boots just fine (but dom0 is not happy about that). 164MB initrd failed too, but 13MB started ok.
Just in case, I tried skipping XHCI console, but it didn't change anything.Host has 16GB of memory, and there is no dom0_mem= parameter. Xen is started from GRUB, using MB2+EFI.
Quoted 6 lines
Hmm, yes, there's an ordering issue: Of course we free initrd space (as used for passing from the boot loader to Xen) only after copying to the designated guest area. Yet
dom0_compute_nr_pages(), intentionally, includes the space in its calculation (adding initial_images_nrpages()'s return value). PV Dom0 isn't affected because to load huge initrd there, the kernel has to request the initrd to not be mapped into the initial allocation.Right, on PV dom0 we do not copy the image to a new set of pages, we simply assign the pages where the initrd resides to the domain. We can't populate those pages in the p2m as-is, otherwise we would shatter super pages.
I think the fix below should do it, it's likely the best we can do. Can you please give it a try Marek?
Thanks, Roger.
---xen/arch/x86/dom0_build.c +0
-0
xen/arch/x86/dom0_build.c unsigned long __init dom0_compute_nr_pages( · line 349
Quoted 8 lines
xen/arch/x86/dom0_build.c +1
-1@@ -343,7 +343,7 @@ unsigned long __init dom0_compute_nr_pages(for_each_node_mask ( node, dom0_nodes ) avail += avail_domheap_pages_region(node, 0, 0) + -initial_images_nrpages (node); +is_pv_domain (d) ?initial_images_nrpages (node) :0 ;/* Reserve memory for further dom0 vcpu-struct allocations... */ avail -= (d->max_vcpus -1UL )
I'm working on a more complex patch, that attempts to account the memory used by the init images towards the reserved amount that's kept by Xen. This should make accounting a bit better, in that we won't end up reserving the Xen memory plus the memory used by the init images.
It's still however a WIP, but would you mind giving it a try?
Thanks, Roger.
---
xen/arch/x86/dom0_build.c +31 -7
@@ -325,10 +325,18 @@ unsigned long __init dom0_paging_pages(const struct domain *d,* If allocation isn't specified, reserve 1/16th of available memory for * things like DMA buffers. This reservation is clamped to a maximum of 128MB. */ -static unsigned long __init default_nr_pages (unsigned long avail) +static unsigned long __init default_nr_pages (unsigned long avail, +unsigned long init_images) { -return avail - (pv_shim ?pv_shim_mem (avail) - :min (avail /16 ,128UL << (20 -PAGE_SHIFT ))); +unsigned long rsvd =min (avail /16 ,128UL << (20 -PAGE_SHIFT )); + +/* +* Account for memory consumed by initial images as if it was part of the +* reserved amount. +*/ + rsvd -= rsvd <= init_images ? rsvd : init_images; + +return avail - (pv_shim ?pv_shim_mem (avail) : rsvd); } unsigned long __init dom0_compute_nr_pages(
@@ -336,14 +344,28 @@ unsigned long __init dom0_compute_nr_pages( {nodeid_t node;unsigned long avail =0 , nr_pages, min_pages, max_pages, iommu_pages =0 ; +unsigned long init_images =0 ;/* The ordering of operands is to work around a clang5 issue. */ if (CONFIG_DOM0_MEM [0 ] && !dom0_mem_set )parse_dom0_mem (CONFIG_DOM0_MEM ); for_each_node_mask ( node, dom0_nodes ) - avail += avail_domheap_pages_region(node, 0, 0) + -initial_images_nrpages (node); + { + avail +=avail_domheap_pages_region (node,0 ,0 ); + init_images +=initial_images_nrpages (node); + } + +if (is_pv_domain (d) ) + { +/* +* For PV domains the initrd pages are directly assigned to the +* guest, and hence the initrd size counts as free memory that can +* be used by the domain. Set to 0 to prevent further adjustments. +*/ + avail += init_images; + init_images =0 ; + }/* Reserve memory for further dom0 vcpu-struct allocations... */ avail -= (d->max_vcpus -1UL )
@@ -367,7 +389,8 @@ unsigned long __init dom0_compute_nr_pages( {unsigned long cpu_pages; - nr_pages =get_memsize (&dom0_size, avail) ?:default_nr_pages (avail); + nr_pages =get_memsize (&dom0_size, avail) ?: +default_nr_pages (avail, init_images);/* * Clamp according to min/max limits and available memory
@@ -385,7 +408,8 @@ unsigned long __init dom0_compute_nr_pages( avail -= cpu_pages - iommu_pages; } - nr_pages =get_memsize (&dom0_size, avail) ?:default_nr_pages (avail); + nr_pages =get_memsize (&dom0_size, avail) ?: +default_nr_pages (avail, init_images); min_pages =get_memsize (&dom0_min_size, avail); max_pages =get_memsize (&dom0_max_size, avail);
Quoted 68 lines · Roger Pau Monné
Quoted 52 lines · Roger Pau Monné
Quoted 29 lines · Jan Beulich
Quoted 20 lines · Marek Marczykowski-Górecki
Hi,
After fixing the xhci crash, I hit another issue - booting with 236MB initrd doesn't work, I get:
(XEN) [ 3.151856] *** Building a PVH Dom0 *** ... (XEN) [ 3.593940] Unable to allocate memory with order 0! (XEN) [ 3.597110] Failed to setup Dom0 physical memory map (XEN) [ 3.599884] (XEN) [ 3.602482] **************************************** (XEN) [ 3.605272] Panic on CPU 0: (XEN) [ 3.607928] Could not construct d0 (XEN) [ 3.610692] **************************************** (XEN) [ 3.613463] (XEN) [ 3.616035] Reboot in five seconds... (XEN) [ 8.626565] Resetting with ACPI MEMORY or I/O RESET_REG.Full console log: https://gist.github.com/marmarek/c9dbc87bf07b76f2899781755762f565
Quoted 7 lines
If I skip initrd, then it boots just fine (but dom0 is not happy about that). 164MB initrd failed too, but 13MB started ok.
Just in case, I tried skipping XHCI console, but it didn't change anything.Host has 16GB of memory, and there is no dom0_mem= parameter. Xen is started from GRUB, using MB2+EFI.
Quoted 6 lines
Hmm, yes, there's an ordering issue: Of course we free initrd space (as used for passing from the boot loader to Xen) only after copying to the designated guest area. Yet
dom0_compute_nr_pages(), intentionally, includes the space in its calculation (adding initial_images_nrpages()'s return value). PV Dom0 isn't affected because to load huge initrd there, the kernel has to request the initrd to not be mapped into the initial allocation.Right, on PV dom0 we do not copy the image to a new set of pages, we simply assign the pages where the initrd resides to the domain. We can't populate those pages in the p2m as-is, otherwise we would shatter super pages.
I think the fix below should do it, it's likely the best we can do. Can you please give it a try Marek?
Thanks, Roger.
---xen/arch/x86/dom0_build.c +0
-0xen/arch/x86/dom0_build.c unsigned long __init dom0_compute_nr_pages( · line 349
Quoted 8 lines
xen/arch/x86/dom0_build.c +1
-1@@ -343,7 +343,7 @@ unsigned long __init dom0_compute_nr_pages(for_each_node_mask ( node, dom0_nodes ) avail += avail_domheap_pages_region(node, 0, 0) + -initial_images_nrpages (node); +is_pv_domain (d) ?initial_images_nrpages (node) :0 ;/* Reserve memory for further dom0 vcpu-struct allocations... */ avail -= (d->max_vcpus -1UL )I'm working on a more complex patch, that attempts to account the memory used by the init images towards the reserved amount that's kept by Xen. This should make accounting a bit better, in that we won't end up reserving the Xen memory plus the memory used by the init images.
Quoted 1 line
It's still however a WIP, but would you mind giving it a try?
This one worked :)
(XEN) [ 4.014243] Dom0 memory allocation stats:
(XEN) [ 4.017378] order 0 allocations: 4
(XEN) [ 4.020111] order 1 allocations: 3
(XEN) [ 4.022758] order 2 allocations: 4
(XEN) [ 4.025406] order 3 allocations: 4
(XEN) [ 4.028150] order 4 allocations: 3
(XEN) [ 4.030886] order 5 allocations: 2
(XEN) [ 4.033602] order 6 allocations: 2
(XEN) [ 4.036305] order 7 allocations: 6
(XEN) [ 4.039020] order 8 allocations: 6
(XEN) [ 4.041597] order 9 allocations: 5
(XEN) [ 4.044261] order 10 allocations: 9
(XEN) [ 4.046831] order 11 allocations: 7
(XEN) [ 4.049493] order 12 allocations: 9
(XEN) [ 4.052147] order 13 allocations: 7
(XEN) [ 4.054799] order 14 allocations: 6
(XEN) [ 4.057447] order 15 allocations: 7
(XEN) [ 4.060080] order 16 allocations: 7
(XEN) [ 4.062609] order 17 allocations: 5
(XEN) [ 4.065227] order 18 allocations: 9
(XEN) [ 4.921719] ELF: phdr: paddr=0x200000 memsz=0x1ff3928
(XEN) [ 4.924403] ELF: phdr: paddr=0x2200000 memsz=0x1c00000
(XEN) [ 4.927079] ELF: memory: 0x200000 -> 0x3e00000
(XEN) [ 4.929759] ELF: note: PHYS32_RELOC align: 0x200000 min: 0x200000 max: 0x3fffffff
(XEN) [ 4.932884] ELF: note: PHYS32_ENTRY = 0x16a2ca0
(XEN) [ 4.935921] ELF: note: GUEST_OS = "linux"
(XEN) [ 4.938953] ELF: note: GUEST_VERSION = "2.6"
(XEN) [ 4.942005] ELF: note: XEN_VERSION = "xen-3.0"
(XEN) [ 4.945077] ELF: note: VIRT_BASE = 0xffffffff80000000
(XEN) [ 4.948133] ELF: note: INIT_P2M = 0x8000000000
(XEN) [ 4.951203] ELF: note: ENTRY = 0xffffffff82d3c160
(XEN) [ 4.954221] ELF: note: FEATURES = "!writable_page_tables"
(XEN) [ 4.957229] ELF: note: PAE_MODE = "yes"
(XEN) [ 4.960175] ELF: note: L1_MFN_VALID
(XEN) [ 4.962775] ELF: note: MOD_START_PFN = 0x1
(XEN) [ 4.965675] ELF: note: PADDR_OFFSET = 0
(XEN) [ 4.968540] ELF: note: SUPPORTED_FEATURES = 0x8801
(XEN) [ 4.971420] ELF: note: LOADER = "generic"
(XEN) [ 4.974303] ELF: note: SUSPEND_CANCEL = 0x1
(XEN) [ 4.977186] ELF: Found PVH image
(XEN) [ 4.979910] ELF: addresses:
(XEN) [ 4.982237] virt_base = 0x0
(XEN) [ 4.984694] elf_paddr_offset = 0x0
(XEN) [ 4.987141] virt_offset = 0x0
(XEN) [ 4.989599] virt_kstart = 0x200000
(XEN) [ 4.992044] virt_kend = 0x3e00000
(XEN) [ 4.994498] virt_entry = 0x16a2ca0
(XEN) [ 4.996966] p2m_base = 0x8000000000
(XEN) [ 4.999415] ELF: phdr 0 at 0x200000 -> 0x21f3928
(XEN) [ 5.007160] ELF: phdr 1 at 0x2200000 -> 0x3e00000
(XEN) [ 5.055448] Dom0 memory map:
(XEN) [ 5.057763] [0000000000000000, 000000000009efff] (usable)
(XEN) [ 5.060281] [000000000009f000, 00000000000fffff] (reserved)
(XEN) [ 5.062812] [0000000000100000, 000000005471afff] (usable)
(XEN) [ 5.065324] [000000005471b000, 000000005475bfff] (reserved)
(XEN) [ 5.067891] [000000005475c000, 0000000063c2dfff] (usable)
(XEN) [ 5.070446] [0000000063c2e000, 000000006d17afff] (reserved)
(XEN) [ 5.073036] [000000006d17b000, 000000006d22bfff] (ACPI data)
(XEN) [ 5.075649] [000000006d22c000, 000000006d2ebfff] (ACPI NVS)
(XEN) [ 5.078267] [000000006d2ec000, 000000006fffefff] (reserved)
(XEN) [ 5.080908] [000000006ffff000, 000000006ffffdcb] (usable)
(XEN) [ 5.083560] [000000006ffffdcc, 000000006ffffe97] (ACPI data)
(XEN) [ 5.086243] [0000000070000000, 00000000807fffff] (reserved)
(XEN) [ 5.088943] [00000000c0000000, 00000000cfffffff] (reserved)
(XEN) [ 5.091662] [00000000fe000000, 00000000fe010fff] (reserved)
(XEN) [ 5.094413] [00000000fec00000, 00000000fec00fff] (reserved)
(XEN) [ 5.097193] [00000000fed00000, 00000000fed00fff] (reserved)
(XEN) [ 5.099976] [00000000fed20000, 00000000fed7ffff] (reserved)
(XEN) [ 5.102779] [00000000fee00000, 00000000fee00fff] (reserved)
(XEN) [ 5.105615] [00000000ff000000, 00000001023fffff] (reserved)
(XEN) [ 5.108487] [0000000102400000, 000000045c89cfff] (usable)
(XEN) [ 5.111354] [000000045c89d000, 000000047f7fffff] (unusable)
(XEN) [ 5.114263] Initial low memory virq threshold set at 0x4000 pages.
(XEN) [ 5.117108] Scrubbing Free RAM in backgroundSignature · 3 lines
Best Regards, Marek Marczykowski-Górecki Invisible Things Lab
Quoted 18 lines · Roger Pau Monné
xen/arch/x86/dom0_build.c +6
-2@@ -336,14 +344,28 @@ unsigned long __init dom0_compute_nr_pages( {nodeid_t node;unsigned long avail =0 , nr_pages, min_pages, max_pages, iommu_pages =0 ; +unsigned long init_images =0 ;/* The ordering of operands is to work around a clang5 issue. */ if (CONFIG_DOM0_MEM [0 ] && !dom0_mem_set )parse_dom0_mem (CONFIG_DOM0_MEM ); for_each_node_mask ( node, dom0_nodes ) - avail += avail_domheap_pages_region(node, 0, 0) + -initial_images_nrpages (node); + { + avail +=avail_domheap_pages_region (node,0 ,0 ); + init_images +=initial_images_nrpages (node); + } +
xen/arch/x86/dom0_build.c unsigned long __init dom0_compute_nr_pages( · line 368
Quoted 10 lines
xen/arch/x86/dom0_build.c +10
-0+if (is_pv_domain (d) ) + { +/* +* For PV domains the initrd pages are directly assigned to the +* guest, and hence the initrd size counts as free memory that can +* be used by the domain. Set to 0 to prevent further adjustments. +*/ + avail += init_images; + init_images =0 ; + }
Just to mention: It's still only "may be directly assigned", because of the PV32 special requirements.
Jan
Quoted 37 lines · Roger Pau Monné
Quoted 29 lines · Jan Beulich
Quoted 20 lines · Marek Marczykowski-Górecki
Hi,
After fixing the xhci crash, I hit another issue - booting with 236MB initrd doesn't work, I get:
(XEN) [ 3.151856] *** Building a PVH Dom0 *** ... (XEN) [ 3.593940] Unable to allocate memory with order 0! (XEN) [ 3.597110] Failed to setup Dom0 physical memory map (XEN) [ 3.599884] (XEN) [ 3.602482] **************************************** (XEN) [ 3.605272] Panic on CPU 0: (XEN) [ 3.607928] Could not construct d0 (XEN) [ 3.610692] **************************************** (XEN) [ 3.613463] (XEN) [ 3.616035] Reboot in five seconds... (XEN) [ 8.626565] Resetting with ACPI MEMORY or I/O RESET_REG.Full console log: https://gist.github.com/marmarek/c9dbc87bf07b76f2899781755762f565
Quoted 7 lines
If I skip initrd, then it boots just fine (but dom0 is not happy about that). 164MB initrd failed too, but 13MB started ok.
Just in case, I tried skipping XHCI console, but it didn't change anything.Host has 16GB of memory, and there is no dom0_mem= parameter. Xen is started from GRUB, using MB2+EFI.
Quoted 6 lines
Hmm, yes, there's an ordering issue: Of course we free initrd space (as used for passing from the boot loader to Xen) only after copying to the designated guest area. Yet
dom0_compute_nr_pages(), intentionally, includes the space in its calculation (adding initial_images_nrpages()'s return value). PV Dom0 isn't affected because to load huge initrd there, the kernel has to request the initrd to not be mapped into the initial allocation.
Quoted 7 lines
Right, on PV dom0 we do not copy the image to a new set of pages, we simply assign the pages where the initrd resides to the domain. We can't populate those pages in the p2m as-is, otherwise we would shatter super pages.
I think the fix below should do it, it's likely the best we can do. Can you please give it a try Marek?
With this, it's different:
(XEN) [ 4.510345] Dom0 memory allocation stats:
(XEN) [ 4.513110] order 0 allocations: 1
(XEN) [ 4.515847] order 1 allocations: 1
(XEN) [ 4.518593] order 2 allocations: 2
(XEN) [ 4.521329] order 3 allocations: 2
(XEN) [ 4.524076] order 4 allocations: 1
(XEN) [ 4.526799] order 5 allocations: 1
(XEN) [ 4.529518] order 6 allocations: 1
(XEN) [ 4.532118] order 7 allocations: 2
(XEN) [ 4.534807] order 8 allocations: 2
(XEN) [ 4.537483] order 9 allocations: 1
(XEN) [ 4.540052] order 10 allocations: 2
(XEN) [ 4.542618] order 11 allocations: 2
(XEN) [ 4.545276] order 12 allocations: 1
(XEN) [ 4.547949] order 13 allocations: 1
(XEN) [ 4.550501] order 14 allocations: 1
(XEN) [ 4.553147] order 15 allocations: 1
(XEN) [ 5.393487] ELF: phdr: paddr=0x200000 memsz=0x1ff3928
(XEN) [ 5.396196] ELF: phdr: paddr=0x2200000 memsz=0x1c00000
(XEN) [ 5.398891] ELF: memory: 0x200000 -> 0x3e00000
(XEN) [ 5.401592] ELF: note: PHYS32_RELOC align: 0x200000 min: 0x200000 max: 0x3fffffff
(XEN) [ 5.404632] ELF: note: PHYS32_ENTRY = 0x16a2ca0
(XEN) [ 5.407695] ELF: note: GUEST_OS = "linux"
(XEN) [ 5.410748] ELF: note: GUEST_VERSION = "2.6"
(XEN) [ 5.413810] ELF: note: XEN_VERSION = "xen-3.0"
(XEN) [ 5.416891] ELF: note: VIRT_BASE = 0xffffffff80000000
(XEN) [ 5.419976] ELF: note: INIT_P2M = 0x8000000000
(XEN) [ 5.423062] ELF: note: ENTRY = 0xffffffff82d3c160
(XEN) [ 5.426155] ELF: note: FEATURES = "!writable_page_tables"
(XEN) [ 5.429260] ELF: note: PAE_MODE = "yes"
(XEN) [ 5.432343] ELF: note: L1_MFN_VALID
(XEN) [ 5.434952] ELF: note: MOD_START_PFN = 0x1
(XEN) [ 5.437975] ELF: note: PADDR_OFFSET = 0
(XEN) [ 5.440928] ELF: note: SUPPORTED_FEATURES = 0x8801
(XEN) [ 5.443871] ELF: note: LOADER = "generic"
(XEN) [ 5.446785] ELF: note: SUSPEND_CANCEL = 0x1
(XEN) [ 5.449696] ELF: Found PVH image
(XEN) [ 5.452574] ELF: addresses:
(XEN) [ 5.455002] virt_base = 0x0
(XEN) [ 5.457569] elf_paddr_offset = 0x0
(XEN) [ 5.460102] virt_offset = 0x0
(XEN) [ 5.462569] virt_kstart = 0x200000
(XEN) [ 5.465049] virt_kend = 0x3e00000
(XEN) [ 5.467515] virt_entry = 0x16a2ca0
(XEN) [ 5.469983] p2m_base = 0x8000000000
(XEN) [ 5.472465] ELF: phdr 0 at 0x200000 -> 0x21f3928
(XEN) [ 5.480548] ELF: phdr 1 at 0x2200000 -> 0x3e00000
(XEN) [ 5.487915] Unable to find a memory region to load initrd and metadata
(XEN) [ 5.490466] Failed to load Dom0 kernel
(XEN) [ 5.492954]
(XEN) [ 5.495266] ****************************************
(XEN) [ 5.497755] Panic on CPU 0:
(XEN) [ 5.500065] Could not construct d0
(XEN) [ 5.502470] ****************************************
(XEN) [ 5.504897]
(XEN) [ 5.507134] Reboot in five seconds... (XEN) [ 10.517191] Resetting with ACPI MEMORY or I/O RESET_REG.
FWIW memory map is in the full console dump linked earlier.
I'll test the other patch next.
Quoted 15 lines
---
xen/arch/x86/dom0_build.c +1
-1@@ -343,7 +343,7 @@ unsigned long __init dom0_compute_nr_pages(for_each_node_mask ( node, dom0_nodes ) avail += avail_domheap_pages_region(node, 0, 0) + -initial_images_nrpages (node); +is_pv_domain (d) ?initial_images_nrpages (node) :0 ;/* Reserve memory for further dom0 vcpu-struct allocations... */ avail -= (d->max_vcpus -1UL )
Signature · 3 lines
Best Regards, Marek Marczykowski-Górecki Invisible Things Lab
Quoted 37 lines · Roger Pau Monné
Quoted 29 lines · Jan Beulich
Quoted 20 lines · Marek Marczykowski-Górecki
Hi,
After fixing the xhci crash, I hit another issue - booting with 236MB initrd doesn't work, I get:
(XEN) [ 3.151856] *** Building a PVH Dom0 *** ... (XEN) [ 3.593940] Unable to allocate memory with order 0! (XEN) [ 3.597110] Failed to setup Dom0 physical memory map (XEN) [ 3.599884] (XEN) [ 3.602482] **************************************** (XEN) [ 3.605272] Panic on CPU 0: (XEN) [ 3.607928] Could not construct d0 (XEN) [ 3.610692] **************************************** (XEN) [ 3.613463] (XEN) [ 3.616035] Reboot in five seconds... (XEN) [ 8.626565] Resetting with ACPI MEMORY or I/O RESET_REG.Full console log: https://gist.github.com/marmarek/c9dbc87bf07b76f2899781755762f565
Quoted 7 lines
If I skip initrd, then it boots just fine (but dom0 is not happy about that). 164MB initrd failed too, but 13MB started ok.
Just in case, I tried skipping XHCI console, but it didn't change anything.Host has 16GB of memory, and there is no dom0_mem= parameter. Xen is started from GRUB, using MB2+EFI.
Quoted 6 lines
Hmm, yes, there's an ordering issue: Of course we free initrd space (as used for passing from the boot loader to Xen) only after copying to the designated guest area. Yet
dom0_compute_nr_pages(), intentionally, includes the space in its calculation (adding initial_images_nrpages()'s return value). PV Dom0 isn't affected because to load huge initrd there, the kernel has to request the initrd to not be mapped into the initial allocation.
Quoted 6 lines
Right, on PV dom0 we do not copy the image to a new set of pages, we simply assign the pages where the initrd resides to the domain. We can't populate those pages in the p2m as-is, otherwise we would shatter super pages.
I think the fix below should do it, it's likely the best we can do.
That's at best a workaround imo. We definitely can do better, and the bigger the initrd, the more important it may be that we actually do. One option may be to similarly use the pages directly (i.e. by assigning rather than copying), accepting that we may not be able to use large page mappings then for the respective GFN range.
Jan
Quoted 18 lines
Can you please give it a try Marek?
Thanks, Roger.
---xen/arch/x86/dom0_build.c +1
-1@@ -343,7 +343,7 @@ unsigned long __init dom0_compute_nr_pages(for_each_node_mask ( node, dom0_nodes ) avail += avail_domheap_pages_region(node, 0, 0) + -initial_images_nrpages (node); +is_pv_domain (d) ?initial_images_nrpages (node) :0 ;/* Reserve memory for further dom0 vcpu-struct allocations... */ avail -= (d->max_vcpus -1UL )
Quoted 43 lines · Jan Beulich
Quoted 42 lines · Roger Pau Monné
Quoted 29 lines · Jan Beulich
Quoted 20 lines · Marek Marczykowski-Górecki
Hi,
After fixing the xhci crash, I hit another issue - booting with 236MB initrd doesn't work, I get:
(XEN) [ 3.151856] *** Building a PVH Dom0 *** ... (XEN) [ 3.593940] Unable to allocate memory with order 0! (XEN) [ 3.597110] Failed to setup Dom0 physical memory map (XEN) [ 3.599884] (XEN) [ 3.602482] **************************************** (XEN) [ 3.605272] Panic on CPU 0: (XEN) [ 3.607928] Could not construct d0 (XEN) [ 3.610692] **************************************** (XEN) [ 3.613463] (XEN) [ 3.616035] Reboot in five seconds... (XEN) [ 8.626565] Resetting with ACPI MEMORY or I/O RESET_REG.Full console log: https://gist.github.com/marmarek/c9dbc87bf07b76f2899781755762f565
Quoted 7 lines
If I skip initrd, then it boots just fine (but dom0 is not happy about that). 164MB initrd failed too, but 13MB started ok.
Just in case, I tried skipping XHCI console, but it didn't change anything.Host has 16GB of memory, and there is no dom0_mem= parameter. Xen is started from GRUB, using MB2+EFI.
Quoted 6 lines
Hmm, yes, there's an ordering issue: Of course we free initrd space (as used for passing from the boot loader to Xen) only after copying to the designated guest area. Yet
dom0_compute_nr_pages(), intentionally, includes the space in its calculation (adding initial_images_nrpages()'s return value). PV Dom0 isn't affected because to load huge initrd there, the kernel has to request the initrd to not be mapped into the initial allocation.Right, on PV dom0 we do not copy the image to a new set of pages, we simply assign the pages where the initrd resides to the domain. We can't populate those pages in the p2m as-is, otherwise we would shatter super pages.
Quoted 4 lines
Quoted 1 line
I think the fix below should do it, it's likely the best we can do.
That's at best a workaround imo. We definitely can do better, and the bigger the initrd, the more important it may be that we actually do.
See the second patch I gave to Marek. That one is slightly better by accounting for the initial images space as part of the reserved space.
Quoted 4 lines
One option may
be to similarly use the pages directly (i.e. by assigning rather than copying), accepting that we may not be able to use large page mappings then for the respective GFN range.
Hm, there's always going to be a trade-off. I think I would prefer having 1G pages in the p2m, rather than a bit more memory due to direct adding the initrd into the p2m.
Thanks, Roger.
Quoted 44 lines · Roger Pau Monné
Quoted 43 lines · Jan Beulich
Quoted 42 lines · Roger Pau Monné
Quoted 29 lines · Jan Beulich
Quoted 20 lines · Marek Marczykowski-Górecki
Hi,
After fixing the xhci crash, I hit another issue - booting with 236MB initrd doesn't work, I get:
(XEN) [ 3.151856] *** Building a PVH Dom0 *** ... (XEN) [ 3.593940] Unable to allocate memory with order 0! (XEN) [ 3.597110] Failed to setup Dom0 physical memory map (XEN) [ 3.599884] (XEN) [ 3.602482] **************************************** (XEN) [ 3.605272] Panic on CPU 0: (XEN) [ 3.607928] Could not construct d0 (XEN) [ 3.610692] **************************************** (XEN) [ 3.613463] (XEN) [ 3.616035] Reboot in five seconds... (XEN) [ 8.626565] Resetting with ACPI MEMORY or I/O RESET_REG.Full console log: https://gist.github.com/marmarek/c9dbc87bf07b76f2899781755762f565
Quoted 7 lines
If I skip initrd, then it boots just fine (but dom0 is not happy about that). 164MB initrd failed too, but 13MB started ok.
Just in case, I tried skipping XHCI console, but it didn't change anything.Host has 16GB of memory, and there is no dom0_mem= parameter. Xen is started from GRUB, using MB2+EFI.
Quoted 6 lines
Hmm, yes, there's an ordering issue: Of course we free initrd space (as used for passing from the boot loader to Xen) only after copying to the designated guest area. Yet
dom0_compute_nr_pages(), intentionally, includes the space in its calculation (adding initial_images_nrpages()'s return value). PV Dom0 isn't affected because to load huge initrd there, the kernel has to request the initrd to not be mapped into the initial allocation.Right, on PV dom0 we do not copy the image to a new set of pages, we simply assign the pages where the initrd resides to the domain. We can't populate those pages in the p2m as-is, otherwise we would shatter super pages.
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
I think the fix below should do it, it's likely the best we can do.
That's at best a workaround imo. We definitely can do better, and the bigger the initrd, the more important it may be that we actually do.
See the second patch I gave to Marek. That one is slightly better by accounting for the initial images space as part of the reserved space.
Will check; didn't make it there, yet.
Quoted 8 lines
Quoted 4 lines
One option may
be to similarly use the pages directly (i.e. by assigning rather than copying), accepting that we may not be able to use large page mappings then for the respective GFN range.Hm, there's always going to be a trade-off. I think I would prefer having 1G pages in the p2m, rather than a bit more memory due to direct adding the initrd into the p2m.
"A bit more" may not get it, when considering e.g. a 2Gb initrd on a 4Gb system. And of course "use pages directly" is only the simplest of the possible approaches. We could "shift" the initrd some, or we could make sure to allocate a 2Mb or 1Gb aligned region to hold it right from the beginning.
Jan
Quoted 16 lines · Marek Marczykowski-Górecki
After fixing the xhci crash, I hit another issue - booting with 236MB initrd doesn't work, I get:
(XEN) [ 3.151856] *** Building a PVH Dom0 *** ... (XEN) [ 3.593940] Unable to allocate memory with order 0! (XEN) [ 3.597110] Failed to setup Dom0 physical memory map (XEN) [ 3.599884] (XEN) [ 3.602482] **************************************** (XEN) [ 3.605272] Panic on CPU 0: (XEN) [ 3.607928] Could not construct d0 (XEN) [ 3.610692] **************************************** (XEN) [ 3.613463] (XEN) [ 3.616035] Reboot in five seconds... (XEN) [ 8.626565] Resetting with ACPI MEMORY or I/O RESET_REG.
Quoted 6 lines
Full console log: https://gist.github.com/marmarek/c9dbc87bf07b76f2899781755762f565
If I skip initrd, then it boots just fine (but dom0 is not happy about that). 164MB initrd failed too, but 13MB started ok.
Just in case, I tried skipping XHCI console, but it didn't change anything.
You may want to give [1] a try, minimally up to patch 11 (patches 01 and 02 are unrelated, so could be omitted as well).
Quoted 1 line
Interestingly, this appear to have worked on Xen 4.19.3.
If so, then - judging from the changes which needed making - only out of luck.
Jan
[1] https://lists.xen.org/archives/html/xen-devel/2026-10/msg00491.html
Quoted 19 lines · Jan Beulich
Quoted 18 lines · Marek Marczykowski-Górecki
After fixing the xhci crash, I hit another issue - booting with 236MB initrd doesn't work, I get:
(XEN) [ 3.151856] *** Building a PVH Dom0 *** ... (XEN) [ 3.593940] Unable to allocate memory with order 0! (XEN) [ 3.597110] Failed to setup Dom0 physical memory map (XEN) [ 3.599884] (XEN) [ 3.602482] **************************************** (XEN) [ 3.605272] Panic on CPU 0: (XEN) [ 3.607928] Could not construct d0 (XEN) [ 3.610692] **************************************** (XEN) [ 3.613463] (XEN) [ 3.616035] Reboot in five seconds... (XEN) [ 8.626565] Resetting with ACPI MEMORY or I/O RESET_REG.Full console log: https://gist.github.com/marmarek/c9dbc87bf07b76f2899781755762f565
Quoted 7 lines
Quoted 4 lines
If I skip initrd, then it boots just fine (but dom0 is not happy about that). 164MB initrd failed too, but 13MB started ok.
Just in case, I tried skipping XHCI console, but it didn't change anything.You may want to give [1] a try, minimally up to patch 11 (patches 01 and 02 are unrelated, so could be omitted as well).
I've tested with the full series, and yes, it does fix booting PVH dom0 on this hardware. The console log looks like this now:
https://gist.github.com/marmarek/7a418274520d66468908a1f1ff56aede
Thanks!
Signature · 3 lines
Best Regards, Marek Marczykowski-Górecki Invisible Things Lab