: Allow Xen to boot/run on large memory (>64G) machines
Subsystems (2)
PATCH 1/1 : Allow Xen to boot/run on large memory (>64G) machines
All,
I've been tracking down a problem where dom0 refuses to boot on very large
memory x86_64 machines. Here's what happens:
The hypervisor starts up with 1GB in the DMA zone. Two large allocations come out of the DMA zone; the frame table (in init_frametable()), and the memory for dom0 (in construct_dom0()). With a lot of memory in the box, most of the DMA zone gets allocated during init_frametable; so much so, in fact, that there is no room to make the allocation in construct_dom0, and the dom0 fails to boot with:
(XEN) ****************************************
(XEN) Panic on CPU 0:
(XEN) Not enough RAM for domain 0 allocation.
(XEN) ****************************************
(XEN)
(XEN) Reboot in five seconds...
The solution (suggested by Keir), is to make the frametable allocated out of high memory instead of the DMA zone. The attached patch (against 3.0.3, but the problem is the same in unstable), does this. I tested this out on a 96GB machine; without the patch, the machine would reboot as described above; with the patch, I was able to boot dom0 and create a PV guest with 92GB of memory.
I only compile tested this on ia64, but I don't see anything in it that
should cause problems there.
Note that this is not the end of the story, however. For even larger
machines, it can *still* be the case that the allocation in construct_dom0() fails; in particular, if the order goes above 17, it will fail in the same way.
One way to fix it would be to just allocate that memory out of the normal zone
for x86_64, as well; however, I'm not sure if this will break anything else. Any comments?
arch/x86/mm.c +8 -0
@@ -179,7 +179,15 @@ void __init init_frametable(void)for ( i =0 ; i < nr_pages; i += page_step ) { +#ifdef __x86_64__ +/* for x86_64 we want to allocate the frame table from the top +* of memory rather than the bottom; otherwise, on large memory +* machines (> 64G), we exhaust DMA memory, and dom0 cannot boot +*/ + mfn =alloc_boot_pages_reverse (min (nr_pages - i, page_step), page_step); +#else mfn =alloc_boot_pages (min (nr_pages - i, page_step), page_step); +#endif if ( mfn ==0 )panic ("Not enough memory for frame table\n" ); map_pages_to_xen(
common/page_alloc.c +29 -11
@@ -213,26 +213,44 @@ void init_boot_pages(paddr_t ps, paddr_t } } -unsigned long alloc_boot_pages (unsigned long nr_pfns,unsigned long pfn_align) +static unsigned long check_and_map_page (unsigned long pg,unsigned long nr_pfns) { -unsigned long pg, i; +unsigned long i; -for ( pg =0 ; (pg + nr_pfns) < max_page; pg += pfn_align ) +for ( i =0 ; i < nr_pfns; i++ ) +if (allocated_in_map (pg + i) ) +break ; + +if ( i == nr_pfns ) { -for ( i =0 ; i < nr_pfns; i++ ) -if (allocated_in_map (pg + i) ) -break ; +map_alloc (pg, nr_pfns); +return pg; + } -if ( i == nr_pfns ) - { -map_alloc (pg, nr_pfns); +return 0 ; +} + +unsigned long alloc_boot_pages (unsigned long nr_pfns,unsigned long pfn_align) +{ +unsigned long pg; + +for ( pg =0 ; (pg + nr_pfns) < max_page; pg += pfn_align ) +if (check_and_map_page (pg, nr_pfns))return pg; - } - }return 0 ; } +unsigned long alloc_boot_pages_reverse (unsigned long nr_pfns,unsigned long pfn_align) +{ +unsigned long pg; + +for ( pg = (max_page - nr_pfns); pg >0 ; pg -= pfn_align ) +if (check_and_map_page (pg, nr_pfns)) +return pg; + +return 0 ; +}/*************************
include/xen/mm.h +1 -0
@@ -39,6 +39,7 @@ struct page_info;/* Boot-time allocator. Turns into generic allocator after bootstrap. */ paddr_t init_boot_allocator (paddr_t bitmap_start);void init_boot_pages (paddr_t ps,paddr_t pe); +unsigned long alloc_boot_pages_reverse (unsigned long nr_pfns,unsigned long pfn_align);unsigned long alloc_boot_pages (unsigned long nr_pfns,unsigned long pfn_align);void end_boot_allocator (void );
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel
Quoted 8 lines · Chris Lalancette
Note that this is not the end of the story, however. For even larger
machines, it can *still* be the case that the allocation inconstruct_dom0()fails; in particular, if the order goes above 17, it will fail in the same way.
One way to fix it would be to just allocate that memory out of the normal
zone
forx86_64, as well; however, I'm not sure if this will break anything else. Any comments?
If there are no users of alloc_boot_pages() expecting low memory to be returned then we can adjust the implementation of that existing function rather than introduce a new one.
As for domain_build() there are two considerations: firstly that the allocation is contiguous and secondly that it is from the DMA pool. The builder makes simplifying assumptions based on contiguity. The allocation from DMA pool I think I've tried to get rid of before -- I think I was scuppered by something as simple as the PAE pgdir needing to be allocated from low memory. I think we can stop allocating from the DMA pool, at least for non-PAE host.
-- Keir
Quoted 16 lines
Quoted 1 line
Quoted 1 line
Keir Fraser <[email protected]> 22.02.07 08:50 >>>
Quoted 8 lines · Chris Lalancette
Note that this is not the end of the story, however. For even larger
machines, it can *still* be the case that the allocation inconstruct_dom0()fails; in particular, if the order goes above 17, it will fail in the same way.
One way to fix it would be to just allocate that memory out of the normal
zone
forx86_64, as well; however, I'm not sure if this will break anything else. Any comments?If there are no users of
alloc_boot_pages()expecting low memory to be returned then we can adjust the implementation of that existing function rather than introduce a new one.
Quoted 7 lines
As for
domain_build()there are two considerations: firstly that the allocation is contiguous and secondly that it is from the DMA pool. The builder makes simplifying assumptions based on contiguity. The allocation from DMA pool I think I've tried to get rid of before -- I think I was scuppered by something as simple as the PAE pgdir needing to be allocated from low memory. I think we can stop allocating from the DMA pool, at least for non-PAE host.
The page allocator changes that I posted a while back probably haven't been looked at so fat, given the above comments. The patch that kills the DMA pool changes x86-64 to allocate the dom0 memory without restriction (i386 has to remain restricted, yet not because of DMA address issues, but in order to be able to see the memory in the 1:1 mapping).
Likewise, I'm not certain the changes presented here make a lot of sense in the context of the elimination of the DMA pool and the resulting desire to unify xen heap and domain heap on x86-64.
Jan
Jan
Quoted 5 lines · Jan Beulich
The page allocator changes that I posted a while back probably haven't been looked at so fat, given the above comments. The patch that kills the DMA pool changes x86-64 to allocate the dom0 memory without restriction (i386 has to remain restricted, yet not because of DMA address issues, but in order to be able to see the memory in the 1:1 mapping).
Oh, good point. :-) And it sounds like it makes sense to leave this alone for now and leave it to your patches.
Quoted 3 lines
Likewise, I'm not certain the changes presented here make a lot of sense in the context of the elimination of the DMA pool and the resulting desire to unify xen heap and domain heap on x86-64.
It makes sense for the boot allocator to prefer to allocate from high memory if it can, rather then using what is currently the DMA pool (and, after your patches are applied, will be from relatively-narrow-address-width pools). So I think this patch is good and narrow enough in scope to go straight in (although I think the behaviour of alloc_boot_pages() should be changed rather than adding a new allocator function).
-- Keir
Quoted 10 lines · Keir Fraser
Quoted 5 lines · Jan Beulich
The page allocator changes that I posted a while back probably haven't been looked at so fat, given the above comments. The patch that kills the DMA pool changes x86-64 to allocate the dom0 memory without restriction (i386 has to remain restricted, yet not because of DMA address issues, but in order to be able to see the memory in the 1:1 mapping).
Quoted 3 lines
Oh, good point. :-) And it sounds like it makes sense to leave this alone for now and leave it to your patches.
Ah, OK. That will address the second concern. Jan's right, I didn't actually look at those patches. Thanks for pointing them out.
Quoted 7 lines
It makes sense for the boot allocator to prefer to allocate from high memory if it can, rather then using what is currently the DMA pool (and, after your patches are applied, will be from relatively-narrow-address-width pools). So I think this patch is good and narrow enough in scope to go straight in (although I think the behaviour of
alloc_boot_pages()should be changed rather than adding a new allocator function).
Yeah, I wasn't quite sure how far to go with this. The frame table was the worst offender, so I just went after that. I can whip up a quick patch and test it out here, changing the alloc_boot_pages() to always allocate from the top.
By the way, I assume we only want to do this for x86_64, yes?
Chris Lalancette
Quoted 11 lines · Chris Lalancette
Quoted 6 lines
It makes sense for the boot allocator to prefer to allocate from high memory if it can, rather then using what is currently the DMA pool (and, after your patches are applied, will be from relatively-narrow-address-width pools). So I think this patch is good and narrow enough in scope to go straight in (although I think the behaviour of
alloc_boot_pages()should be changed rather than adding a new allocator function).Yeah, I wasn't quite sure how far to go with this. The frame table was the worst offender, so I just went after that. I can whip up a quick patch and test
it out here, changing thealloc_boot_pages()to always allocate from the top.
Turns out there is one place that wants to allocate from the bottom (the kdump path in arch/x86/setup.c). So I renamed alloc_boot_pages() to alloc_lowmem_boot_pages() and called the new function alloc_boot_pages(). If you care you use the more specific (lowmem) one. I'm about to check in the revised patch.
Quoted 1 line
By the way, I assume we only want to do this for
x86_64, yes?
No, it makes sense to conserve 'narrower' addresses.
-- Keir