Skip to contentSkip to conversation

discontig patch question

DISCUSSION Van Maren, Kevin 3 participants 10 messages Nov 10, 2003 – Nov 12, 2003

lore
Share & tools
lore
#0 lore raw

Hi,

I tried running the discontig patch on a more "normal" machine (one that wasn't fully loaded with memory), and I found the results suprising.

The EFI memory map is simple, and looks like:
0- 4G Node 0 (2G + 2G hole)
4- 8G Node 1
8-12G Node 2
12-16G Node 3
16-20G Node 0 (2 G memory-map I/O reclaim)
with 4G per node, 16GB total.

Because of ORDERROUNDDOWN in count_pages (arch/ia64/mm/init.c), the memory ended up being assigned like this:

0- 8G Node 1 (6G, 2GB hole)
8-16G Node 3 (8G)
16-20G Node 0 (2G)

       Node 2 (0G)

Which was not at all what I wanted.

ORDERROUNDDOWN causes the kernel to assign all memory starting at the (PAGE_SIZE << MAX_ORDER) boundary to the current node, which in my case is 16KB << 19 (hard-coded for IA64), or 8GB.

I understand the GRANULE rounding, but is there a compelling reason that we need 8GB node chunks on IA64 Linux (with 16KB pages)?

Thanks,
Kevin Van Maren

#1 lore raw
Quoted 11 lines · Kevin

The EFI memory map is simple, and looks like:
0- 4G Node 0 (2G + 2G hole)
4- 8G Node 1
8-12G Node 2
12-16G Node 3
16-20G Node 0 (2 G memory-map I/O reclaim)
with 4G per node, 16GB total.

Because of ORDERROUNDDOWN in count_pages (arch/ia64/mm/init.c), the memory ended up being assigned like this:

Quoted 6 lines

0- 8G Node 1 (6G, 2GB hole)
8-16G Node 3 (8G)
16-20G Node 0 (2G)

       Node 2 (0G)

Which was not at all what I wanted.

I guess I didn't see this because the nodes on sn2 are so large (64GB).

Quoted 3 lines

ORDERROUNDDOWN causes the kernel to assign all memory starting at the (PAGE_SIZE << MAX_ORDER) boundary to the current node, which in my case is 16KB << 19 (hard-coded for IA64), or 8GB.

I wonder if that shouldn't be simply 1UL<<MAX_ORDER... That's all that mm/page_alloc.c seems to care about.

Quoted 2 lines

I understand the GRANULE rounding, but is there a compelling reason that we need 8GB node chunks on IA64 Linux (with 16KB pages)?

I don't think so.

Jesse

#2 lore raw
In reply to Van Maren, Kevin
Quoted 19 lines

From: Jesse Barnes [mailto:[email protected]]

Quoted 16 lines · Kevin

The EFI memory map is simple, and looks like:
0- 4G Node 0 (2G + 2G hole)
4- 8G Node 1
8-12G Node 2
12-16G Node 3
16-20G Node 0 (2 G memory-map I/O reclaim)
with 4G per node, 16GB total.

Because of ORDERROUNDDOWN in count_pages (arch/ia64/mm/init.c), the memory ended up being assigned like this:

0- 8G Node 1 (6G, 2GB hole)
8-16G Node 3 (8G)
16-20G Node 0 (2G)

       Node 2 (0G)
Quoted 4 lines
Quoted 1 line

Which was not at all what I wanted.

I guess I didn't see this because the nodes on sn2 are so large (64GB).

I've never run with so little memory before either :-(

Quoted 7 lines
Quoted 3 lines

ORDERROUNDDOWN causes the kernel to assign all memory starting at the (PAGE_SIZE << MAX_ORDER) boundary to the current node, which in my case is 16KB << 19 (hard-coded for IA64), or 8GB.

I wonder if that shouldn't be simply 1UL<<MAX_ORDER... That's all that
mm/page_alloc.c seems to care about.

But doesn't it deal with page-sized chunks?

It makes sense if all the memory chunks have to start on a "MAX_ORDER" boundary, but is that really the case? That's pretty restrictive, at least with such a large MAX_ORDER.

Why is MAX_ORDER 19 on IA64?

Quoted 4 lines
Quoted 2 lines

I understand the GRANULE rounding, but is there a compelling reason that we need 8GB node chunks on IA64 Linux (with 16KB pages)?

I don't think so.

Thanks,
Kevin

#3 lore raw
In reply to Van Maren, Kevin
Quoted 5 lines · Kevin
Quoted 3 lines

I wonder if that shouldn't be simply 1UL<<MAX_ORDER... That's all that
mm/page_alloc.c seems to care about.

But doesn't it deal with page-sized chunks?

Yeah, but 1UL<<MAX_ORDER will always be page aligned, right?

Quoted 5 lines

It makes sense if all the memory chunks have to start on a "MAX_ORDER" boundary, but is that really the case? That's pretty restrictive, at least with such a large MAX_ORDER.

Why is MAX_ORDER 19 on IA64?

For hugetlbfs, afaik. But it should only be 18 according to arch/ia64/Kconfig.

Jesse

#4 lore raw
In reply to Van Maren, Kevin
Quoted 7 lines
Quoted 4 lines

Why is MAX_ORDER 19 on IA64?

For hugetlbfs, afaik. But it should only be 18 according to arch/ia64/Kconfig.

That is correct. For the support of 4G hugepage size (when the normal page size is 16K). Kernel actually takes looks at MAX_ORDER -1 (and not MAX_ORDER).

#5 lore raw
In reply to Van Maren, Kevin
Quoted 1 line

Yeah, but 1UL<<MAX_ORDER will always be page aligned, right?

Only if MAX_ORDER >= PAGE_SHIFT.

But page alignment isn't the question: it is already aligned to the 16MB or 64MB granules.

But you are saying that the address doesn't have to be as strict:
even if allocating 2^MAX_ORDER _pages_, the start doesn't have to be aligned at a natural (PAGE_SIZE<<MAX_ORDER) boundary, and that we can change the ORDERROUNDDOWN to not be as aggressive.

But then it also makes sense to have a smaller MAX_ORDER when not using 4GB hugepages? I'm happy with <= 256MB hugepages with 16GB ram, so I guess I'd rather MAX_ORDER was normally smaller, and increased only with very large hugepage pages.

Thanks,
Kevin Van Maren

#6 lore raw
In reply to Van Maren, Kevin
Quoted 6 lines · Kevin
Quoted 1 line

Yeah, but 1UL<<MAX_ORDER will always be page aligned, right?

Only if MAX_ORDER >= PAGE_SHIFT.

But page alignment isn't the question: it is already aligned to the 16MB or 64MB granules.

Right. Spaced out there for a minute...

Quoted 4 lines

But you are saying that the address doesn't have to be as strict:
even if allocating 2^MAX_ORDER _pages_, the start doesn't have to be aligned at a natural (PAGE_SIZE<<MAX_ORDER) boundary, and that we can change the ORDERROUNDDOWN to not be as aggressive.

Well, strictly speaking I don't think start _has_ to align on those conditions, but the hugetlb stuff may that it does (I haven't looked).

Quoted 4 lines

But then it also makes sense to have a smaller MAX_ORDER when not using 4GB hugepages? I'm happy with <= 256MB hugepages with 16GB ram, so I guess I'd rather MAX_ORDER was normally smaller, and increased only with very large hugepage pages.

That makes sense to me. It seems like FORCE_MAX_ZONEORDER should depend on HUGETLB_PAGE_SIZE_* so that we don't apply unnecessary alignment constraints. Of course, there's probably something I'm missing, Rohit might know more.

Jesse

#7 lore raw
In reply to Van Maren, Kevin
Quoted 2 lines · Jesse Barnes

Well, strictly speaking I don't think start _has_ to align on those conditions, but the hugetlb stuff may that it does (I haven't looked).

s/may/may assume/

#8 lore raw
In reply to Van Maren, Kevin

In my opinion since the other zone build functions depend on (1UL << MAX_ORDER -1) for forcing the zone alignment, so the same should be used here in ORDERSDOWN also. If that seems too aggressive for any reason then you should change the ORDERROUNDDOWN minimally to have (PAGE_SIZE << (MAX_ORDER -1)). That should align the modules at 4G boundary for 16K PAGE_SIZE.

Hugetlb needs certain order (of NORMAL_PAGE_SIZE)of contiguous pages to be available for allocation. The start of this allocation needs to be at least HUGE_PAGE_SIZE_ALIGNED. And I think that part is guaranteed by buddy page_allocator(while allocating a certain order of pages).

rohit

Quoted 17 lines

-----Original Message-----
From: Jesse Barnes [mailto:[email protected]]
Sent: Monday, November 10, 2003 11:09 AM
To: Van Maren, Kevin
Cc: Seth, Rohit; [email protected]
Subject: Re: discontig patch question

Quoted 6 lines · Kevin
Quoted 1 line

Yeah, but 1UL<<MAX_ORDER will always be page aligned, right?

Only if MAX_ORDER >= PAGE_SHIFT.

But page alignment isn't the question: it is already aligned to the 16MB or 64MB granules.

Right. Spaced out there for a minute...

Quoted 10 lines
Quoted 4 lines

But you are saying that the address doesn't have to be as strict:
even if allocating 2^MAX_ORDER _pages_, the start doesn't have to be aligned at a natural (PAGE_SIZE<<MAX_ORDER) boundary, and that we can change the ORDERROUNDDOWN to not be as aggressive.

Well, strictly speaking I don't think start _has_ to align on those conditions, but the hugetlb stuff may that it does (I haven't looked).

Quoted 2 lines

But then it also makes sense to have a smaller MAX_ORDER when not using 4GB hugepages? I'm happy with <= 256MB hugepages with 16GB

ram,

Quoted 4 lines
Quoted 2 lines

so I guess I'd rather MAX_ORDER was normally smaller, and increased only with very large hugepage pages.

That makes sense to me. It seems like FORCE_MAX_ZONEORDER should

depend

Quoted 5 lines

on HUGETLB_PAGE_SIZE_* so that we don't apply unnecessary alignment constraints. Of course, there's probably something I'm missing, Rohit might know more.

Jesse

#9 lore raw
In reply to Van Maren, Kevin

The system boots and functions properly with 4GB nodes, using #define ORDERROUNDDOWN(n) ((n) & ~((PAGE_SIZE<<MAX_ORDER-1)-1)) Leaving it out (1GB boundary) the kernel doesn't boot (memory allocation problem). But I'm (obviously) not using 4GB pages.

So either that fix or dropping MAX_ORDER needs to be made.

Kevin Van Maren

Quoted 75 lines

-----Original Message-----
From: [email protected] [mailto:[email protected]]On Behalf Of Seth, Rohit
Sent: Monday, November 10, 2003 1:21 PM
To: Jesse Barnes; Van Maren, Kevin
Cc: [email protected]
Subject: RE: discontig patch question

In my opinion since the other zone build functions depend on (1UL << MAX_ORDER -1) for forcing the zone alignment, so the same should be used
here in ORDERSDOWN also. If that seems too aggressive for any reason then you should change the ORDERROUNDDOWN minimally to have (PAGE_SIZE << (MAX_ORDER -1)). That should align the modules at 4G boundary for 16K PAGE_SIZE.

Hugetlb needs certain order (of NORMAL_PAGE_SIZE)of contiguous pages to
be available for allocation. The start of this allocation needs to be at least HUGE_PAGE_SIZE_ALIGNED. And I think that part is guaranteed by
buddy page_allocator(while allocating a certain order of pages).

rohit

Quoted 17 lines

-----Original Message-----
From: Jesse Barnes [mailto:[email protected]]
Sent: Monday, November 10, 2003 11:09 AM
To: Van Maren, Kevin
Cc: Seth, Rohit; [email protected]
Subject: Re: discontig patch question

Quoted 6 lines · Kevin
Quoted 1 line

Yeah, but 1UL<<MAX_ORDER will always be page aligned, right?

Only if MAX_ORDER >= PAGE_SHIFT.

But page alignment isn't the question: it is already aligned to the 16MB or 64MB granules.

Right. Spaced out there for a minute...

Quoted 7 lines
Quoted 4 lines

But you are saying that the address doesn't have to be as strict:
even if allocating 2^MAX_ORDER _pages_, the start doesn't have to be aligned at a natural (PAGE_SIZE<<MAX_ORDER) boundary, and that we can change the ORDERROUNDDOWN to not be as aggressive.

Well, strictly speaking I don't think start _has_ to align on those conditions, but the hugetlb stuff may that it does (I

haven't looked).

Quoted 3 lines
Quoted 3 lines

But then it also makes sense to have a smaller MAX_ORDER when not using 4GB hugepages? I'm happy with <= 256MB hugepages with 16GB

ram,

Quoted 1 line
Quoted 1 line

so I guess I'd rather MAX_ORDER was normally smaller, and

increased

Quoted 3 lines
Quoted 1 line

only with very large hugepage pages.

That makes sense to me. It seems like FORCE_MAX_ZONEORDER should

depend

Quoted 2 lines

on HUGETLB_PAGE_SIZE_* so that we don't apply unnecessary alignment constraints. Of course, there's probably something I'm

missing, Rohit

Quoted 3 lines

might know more.

Jesse

-
To unsubscribe from this list: send the line "unsubscribe linux-ia64" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html