discontig patch question
Hi,
I tried running the discontig patch on a more "normal" machine (one that wasn't fully loaded with memory), and I found the results suprising.
The EFI memory map is simple, and looks like:
0- 4G Node 0 (2G + 2G hole)
4- 8G Node 1
8-12G Node 2
12-16G Node 3
16-20G Node 0 (2 G memory-map I/O reclaim)
with 4G per node, 16GB total.
Because of ORDERROUNDDOWN in count_pages (arch/ia64/mm/init.c), the memory ended up being assigned like this:
0- 8G Node 1 (6G, 2GB hole)
8-16G Node 3 (8G)
16-20G Node 0 (2G)
Node 2 (0G)
Which was not at all what I wanted.
ORDERROUNDDOWN causes the kernel to assign all memory starting at the (PAGE_SIZE << MAX_ORDER) boundary to the current node, which in my case is 16KB << 19 (hard-coded for IA64), or 8GB.
I understand the GRANULE rounding, but is there a compelling reason that we need 8GB node chunks on IA64 Linux (with 16KB pages)?
Thanks,
Kevin Van Maren
Quoted 11 lines · Kevin
The EFI memory map is simple, and looks like:
0- 4G Node 0 (2G + 2G hole)
4- 8G Node 1
8-12G Node 2
12-16G Node 3
16-20G Node 0 (2 G memory-map I/O reclaim)
with 4G per node, 16GB total.Because of ORDERROUNDDOWN in
count_pages(arch/ia64/mm/init.c), the memory ended up being assigned like this:
Quoted 6 lines
0- 8G Node 1 (6G, 2GB hole)
8-16G Node 3 (8G)
16-20G Node 0 (2G)Node 2 (0G)Which was not at all what I wanted.
I guess I didn't see this because the nodes on sn2 are so large (64GB).
Quoted 3 lines
ORDERROUNDDOWN causes the kernel to assign all memory starting at the (
PAGE_SIZE<<MAX_ORDER) boundary to the current node, which in my case is 16KB << 19 (hard-coded for IA64), or 8GB.
I wonder if that shouldn't be simply 1UL<<MAX_ORDER... That's all that mm/page_alloc.c seems to care about.
Quoted 2 lines
I understand the GRANULE rounding, but is there a compelling reason that we need 8GB node chunks on IA64 Linux (with 16KB pages)?
I don't think so.
Jesse
Quoted 19 lines
From: Jesse Barnes [mailto:[email protected]]
Quoted 16 lines · Kevin
The EFI memory map is simple, and looks like:
0- 4G Node 0 (2G + 2G hole)
4- 8G Node 1
8-12G Node 2
12-16G Node 3
16-20G Node 0 (2 G memory-map I/O reclaim)
with 4G per node, 16GB total.Because of ORDERROUNDDOWN in
count_pages(arch/ia64/mm/init.c), the memory ended up being assigned like this:0- 8G Node 1 (6G, 2GB hole)
8-16G Node 3 (8G)
16-20G Node 0 (2G)Node 2 (0G)
Quoted 4 lines
Quoted 1 line
Which was not at all what I wanted.
I guess I didn't see this because the nodes on sn2 are so large (64GB).
I've never run with so little memory before either :-(
Quoted 7 lines
Quoted 3 lines
ORDERROUNDDOWN causes the kernel to assign all memory starting at the (
PAGE_SIZE<<MAX_ORDER) boundary to the current node, which in my case is 16KB << 19 (hard-coded for IA64), or 8GB.I wonder if that shouldn't be simply 1UL<<MAX_ORDER... That's all that
mm/page_alloc.cseems to care about.
But doesn't it deal with page-sized chunks?
It makes sense if all the memory chunks have to start on a "MAX_ORDER" boundary, but is that really the case? That's pretty restrictive, at least with such a large MAX_ORDER.
Why is MAX_ORDER 19 on IA64?
Quoted 4 lines
Quoted 2 lines
I understand the GRANULE rounding, but is there a compelling reason that we need 8GB node chunks on IA64 Linux (with 16KB pages)?
I don't think so.
Thanks,
Kevin
Quoted 5 lines · Kevin
Quoted 3 lines
I wonder if that shouldn't be simply 1UL<<MAX_ORDER... That's all that
mm/page_alloc.cseems to care about.But doesn't it deal with page-sized chunks?
Yeah, but 1UL<<MAX_ORDER will always be page aligned, right?
Quoted 5 lines
It makes sense if all the memory chunks have to start on a "
MAX_ORDER" boundary, but is that really the case? That's pretty restrictive, at least with such a largeMAX_ORDER.Why is
MAX_ORDER19 on IA64?
For hugetlbfs, afaik. But it should only be 18 according to arch/ia64/Kconfig.
Jesse
Quoted 7 lines
Quoted 4 lines
Why is
MAX_ORDER19 on IA64?For hugetlbfs, afaik. But it should only be 18 according to
arch/ia64/Kconfig.
That is correct. For the support of 4G hugepage size (when the normal page size is 16K). Kernel actually takes looks at MAX_ORDER -1 (and not MAX_ORDER).
Quoted 1 line
Yeah, but 1UL<<MAX_ORDER will always be page aligned, right?
Only if MAX_ORDER >= PAGE_SHIFT.
But page alignment isn't the question: it is already aligned to the 16MB or 64MB granules.
But you are saying that the address doesn't have to be as strict:
even if allocating 2^MAX_ORDER _pages_, the start doesn't have to be aligned at a natural (PAGE_SIZE<<MAX_ORDER) boundary, and that we can change the ORDERROUNDDOWN to not be as aggressive.
But then it also makes sense to have a smaller MAX_ORDER when not using 4GB hugepages? I'm happy with <= 256MB hugepages with 16GB ram, so I guess I'd rather MAX_ORDER was normally smaller, and increased only with very large hugepage pages.
Thanks,
Kevin Van Maren
Quoted 6 lines · Kevin
Quoted 1 line
Yeah, but 1UL<<MAX_ORDER will always be page aligned, right?
Only if
MAX_ORDER>=PAGE_SHIFT.But page alignment isn't the question: it is already aligned to the 16MB or 64MB granules.
Right. Spaced out there for a minute...
Quoted 4 lines
But you are saying that the address doesn't have to be as strict:
even if allocating 2^MAX_ORDER _pages_, the start doesn't have to be aligned at a natural (PAGE_SIZE<<MAX_ORDER) boundary, and that we can change the ORDERROUNDDOWN to not be as aggressive.
Well, strictly speaking I don't think start _has_ to align on those conditions, but the hugetlb stuff may that it does (I haven't looked).
Quoted 4 lines
But then it also makes sense to have a smaller
MAX_ORDERwhen not using 4GB hugepages? I'm happy with <= 256MB hugepages with 16GB ram, so I guess I'd ratherMAX_ORDERwas normally smaller, and increased only with very large hugepage pages.
That makes sense to me. It seems like FORCE_MAX_ZONEORDER should depend on HUGETLB_PAGE_SIZE_* so that we don't apply unnecessary alignment constraints. Of course, there's probably something I'm missing, Rohit might know more.
Jesse
Quoted 2 lines · Jesse Barnes
Well, strictly speaking I don't think start _has_ to align on those conditions, but the hugetlb stuff may that it does (I haven't looked).
s/may/may assume/
In my opinion since the other zone build functions depend on (1UL << MAX_ORDER -1) for forcing the zone alignment, so the same should be used here in ORDERSDOWN also. If that seems too aggressive for any reason then you should change the ORDERROUNDDOWN minimally to have (PAGE_SIZE << (MAX_ORDER -1)). That should align the modules at 4G boundary for 16K PAGE_SIZE.
Hugetlb needs certain order (of NORMAL_PAGE_SIZE)of contiguous pages to be available for allocation. The start of this allocation needs to be at least HUGE_PAGE_SIZE_ALIGNED. And I think that part is guaranteed by buddy page_allocator(while allocating a certain order of pages).
rohit
Quoted 17 lines
-----Original Message-----
From: Jesse Barnes [mailto:[email protected]]
Sent: Monday, November 10, 2003 11:09 AM
To: Van Maren, Kevin
Cc: Seth, Rohit; [email protected]
Subject: Re: discontig patch questionQuoted 6 lines · Kevin
Quoted 1 line
Yeah, but 1UL<<MAX_ORDER will always be page aligned, right?
Only if
MAX_ORDER>=PAGE_SHIFT.But page alignment isn't the question: it is already aligned to the 16MB or 64MB granules.
Right. Spaced out there for a minute...
Quoted 10 lines
Quoted 4 lines
But you are saying that the address doesn't have to be as strict:
even if allocating 2^MAX_ORDER _pages_, the start doesn't have to be aligned at a natural (PAGE_SIZE<<MAX_ORDER) boundary, and that we can change the ORDERROUNDDOWN to not be as aggressive.Well, strictly speaking I don't think start _has_ to align on those conditions, but the hugetlb stuff may that it does (I haven't looked).
Quoted 2 lines
But then it also makes sense to have a smaller
MAX_ORDERwhen not using 4GB hugepages? I'm happy with <= 256MB hugepages with 16GB
ram,
Quoted 4 lines
Quoted 2 lines
so I guess I'd rather
MAX_ORDERwas normally smaller, and increased only with very large hugepage pages.That makes sense to me. It seems like
FORCE_MAX_ZONEORDERshould
depend
Quoted 5 lines
on HUGETLB_PAGE_SIZE_* so that we don't apply unnecessary alignment constraints. Of course, there's probably something I'm missing, Rohit might know more.
Jesse
The system boots and functions properly with 4GB nodes, using #define ORDERROUNDDOWN(n) ((n) & ~((PAGE_SIZE<<MAX_ORDER-1)-1)) Leaving it out (1GB boundary) the kernel doesn't boot (memory allocation problem). But I'm (obviously) not using 4GB pages.
So either that fix or dropping MAX_ORDER needs to be made.
Kevin Van Maren
Quoted 75 lines
-----Original Message-----
From: [email protected] [mailto:[email protected]]On Behalf Of Seth, Rohit
Sent: Monday, November 10, 2003 1:21 PM
To: Jesse Barnes; Van Maren, Kevin
Cc: [email protected]
Subject: RE: discontig patch questionIn my opinion since the other zone build functions depend on (1UL <<
MAX_ORDER-1) for forcing the zone alignment, so the same should be used
here in ORDERSDOWN also. If that seems too aggressive for any reason then you should change the ORDERROUNDDOWN minimally to have (PAGE_SIZE<< (MAX_ORDER-1)). That should align the modules at 4G boundary for 16KPAGE_SIZE.Hugetlb needs certain order (of NORMAL_PAGE_SIZE)of contiguous pages to
be available for allocation. The start of this allocation needs to be at leastHUGE_PAGE_SIZE_ALIGNED. And I think that part is guaranteed by
buddypage_allocator(while allocating a certain order of pages).rohit
Quoted 17 lines
-----Original Message-----
From: Jesse Barnes [mailto:[email protected]]
Sent: Monday, November 10, 2003 11:09 AM
To: Van Maren, Kevin
Cc: Seth, Rohit; [email protected]
Subject: Re: discontig patch questionQuoted 6 lines · Kevin
Quoted 1 line
Yeah, but 1UL<<MAX_ORDER will always be page aligned, right?
Only if
MAX_ORDER>=PAGE_SHIFT.But page alignment isn't the question: it is already aligned to the 16MB or 64MB granules.
Right. Spaced out there for a minute...
Quoted 7 lines
Quoted 4 lines
But you are saying that the address doesn't have to be as strict:
even if allocating 2^MAX_ORDER _pages_, the start doesn't have to be aligned at a natural (PAGE_SIZE<<MAX_ORDER) boundary, and that we can change the ORDERROUNDDOWN to not be as aggressive.Well, strictly speaking I don't think start _has_ to align on those conditions, but the hugetlb stuff may that it does (I
haven't looked).
Quoted 3 lines
Quoted 3 lines
But then it also makes sense to have a smaller
MAX_ORDERwhen not using 4GB hugepages? I'm happy with <= 256MB hugepages with 16GBram,
Quoted 1 line
Quoted 1 line
so I guess I'd rather
MAX_ORDERwas normally smaller, andincreased
Quoted 3 lines
Quoted 1 line
only with very large hugepage pages.
That makes sense to me. It seems like
FORCE_MAX_ZONEORDERshoulddepend
Quoted 2 lines
on HUGETLB_PAGE_SIZE_* so that we don't apply unnecessary alignment constraints. Of course, there's probably something I'm
missing, Rohit
Quoted 3 lines
might know more.
Jesse
-
To unsubscribe from this list: send the line "unsubscribe linux-ia64" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html