Skip to contentSkip to conversation

Large Pages - Linux Foundation HPC

DISCUSSION Badari Pulavarty 5 participants 7 messages Apr 21, 2009 – Apr 28, 2009

lore
Share & tools
lore
#0 lore raw

Hi Dave,

On the Linux foundation HPC track summary, I saw:

-- Memory and interface to it - mapping memory into apps, nodes going down due to memory exhaustion
- large pages important - current state not good enough

What does this mean ? Whats not good enough ? What do they want ? Did they spell out the actual requirements and why ? What are your findings ?

Thanks,
Badari

#1 lore raw
Quoted 6 lines · Badari Pulavarty

Hi Dave,

On the Linux foundation HPC track summary, I saw:

-- Memory and interface to it - mapping memory into apps
- large pages important - current state not good enough

I'm not sure exactly what this means. But, there was continuing concern about large page interfaces. hugetlbfs is fine, but it still requires special tools, planning, and requires some modification of the app. We can modify it with linker tricks or with LD_PRELOAD, but those certainly don't work everywhere. I was told over and over again that hugetlbfs isn't a sufficient interface for large pages, no matter how much userspace we try to stick in front of it.

Some of their apps get a 6-7x speedup from large pages!

Fragmentation also isn't an issue for a big chunk of the users since they reboot between each job.

Quoted 1 line

nodes going down due to memory exhaustion

Virtually all the apps in an HPC environment start up try to use all the memory they can get their hands on. With strict overcommit on, that probably means brk() or mmap() until they fail. They also usually mlock() anything they're able to allocate. Swapping is the devil to them. :)

Basically, what all the apps do is a recipe for stressing the VM and triggering the OOM killer. Most of the users simply hack the kernel and replace the OOM killer with one that fits their needs. Some have an attitude that "the user's app should never die" and others "the user caused this, so kill their app". Basically, there's no way to make everyone happy since they have conflicting requirements. But, this is true of the kernel in general... nothing special here.

The split LRU should help things. It will at least make our memory scanning more efficient and ensure we're making more efficient reclaim progress. I'm not sure that anyone there knew about the oom_adjust and oom_score knobs in /proc. They do now. :)

One of my suggestions was to use the memory resource controller. They could give each app 95% (or whatever) of the system. This should let them keep their current "consume all memory" behavior, but stop them at sane limits.

That leads into another issue, which is the "wedding cake" software stack. There are a lot of software dependencies both in and out of the kernel. It is hard to change individual components, especially in the lower levels. This leads many of the users to use old (think 2.6.9) kernels. Nobody runs mainline, of course.

Then, there's Lustre. Everybody uses it, it's definitely a big hunk of the "wedding cake". I haven't seen any LKML postings on it in years and I really wonder how it interacts with the VM. No idea.

There's a "Hyperion cluster" which is for testing new HPC software on a decently sized cluster. One suggestion of ours was to try and get mainline tested on this every so often to look for regressions since we're not able to glean feedback from 2.6.9 kernel users. We'll see where that goes.

Quoted 1 line

checkpoint/restart

Many of the MPI implementations have mechanisms in userspace for checkpointing of user jobs. Most cluster administrators instruct their users to use these mechanisms. Some do. Most don't.

-- Dave

#2 lore raw

[Fix my email address to [email protected]]

Quoted 29 lines · Dave Hansen
Quoted 6 lines · Badari Pulavarty

Hi Dave,

On the Linux foundation HPC track summary, I saw:

-- Memory and interface to it - mapping memory into apps
- large pages important - current state not good enough

I'm not sure exactly what this means. But, there was continuing concern about large page interfaces. hugetlbfs is fine, but it still requires special tools, planning, and requires some modification of the app. We can modify it with linker tricks or with LD_PRELOAD, but those certainly don't work everywhere. I was told over and over again that hugetlbfs isn't a sufficient interface for large pages, no matter how much userspace we try to stick in front of it.

Some of their apps get a 6-7x speedup from large pages!

Fragmentation also isn't an issue for a big chunk of the users since they reboot between each job.

Quoted 1 line

nodes going down due to memory exhaustion

Virtually all the apps in an HPC environment start up try to use all the memory they can get their hands on. With strict overcommit on, that probably means brk() or mmap() until they fail. They also usually mlock() anything they're able to allocate. Swapping is the devil to them. :)

Quoted 7 lines

Basically, what all the apps do is a recipe for stressing the VM and triggering the OOM killer. Most of the users simply hack the kernel and replace the OOM killer with one that fits their needs. Some have an attitude that "the user's app should never die" and others "the user caused this, so kill their app". Basically, there's no way to make everyone happy since they have conflicting requirements. But, this is true of the kernel in general... nothing special here.

OOM killer has been a hot topic. Have you seen Dan Malek's patches at
http://lkml.org/lkml/2009/4/13/276.

Quoted 5 lines

The split LRU should help things. It will at least make our memory scanning more efficient and ensure we're making more efficient reclaim progress. I'm not sure that anyone there knew about the oom_adjust and oom_score knobs in /proc. They do now. :)

:-)

Quoted 6 lines

One of my suggestions was to use the memory resource controller. They could give each app 95% (or whatever) of the system. This should let them keep their current "consume all memory" behavior, but stop them at sane limits.

Soft limits should help as well, basically we are trying to allow unrestricted memory access until there is contention. The patches are still under development.

Quoted 16 lines

That leads into another issue, which is the "wedding cake" software stack. There are a lot of software dependencies both in and out of the kernel. It is hard to change individual components, especially in the lower levels. This leads many of the users to use old (think 2.6.9) kernels. Nobody runs mainline, of course.

Then, there's Lustre. Everybody uses it, it's definitely a big hunk of the "wedding cake". I haven't seen any LKML postings on it in years and I really wonder how it interacts with the VM. No idea.

There's a "Hyperion cluster" which is for testing new HPC software on a decently sized cluster. One suggestion of ours was to try and get mainline tested on this every so often to look for regressions since we're not able to glean feedback from 2.6.9 kernel users. We'll see where that goes.

Quoted 6 lines
Quoted 1 line

checkpoint/restart

Many of the MPI implementations have mechanisms in userspace for checkpointing of user jobs. Most cluster administrators instruct their users to use these mechanisms. Some do. Most don't.

Good inputs and summary. Thanks!

Signature · 1 line
	Balbir
#3 lore raw
Quoted 23 lines · Balbir Singh

[Fix my email address to [email protected]]

Quoted 20 lines · Dave Hansen
Quoted 6 lines · Badari Pulavarty

Hi Dave,

On the Linux foundation HPC track summary, I saw:

-- Memory and interface to it - mapping memory into apps
- large pages important - current state not good enough

I'm not sure exactly what this means. But, there was continuing concern about large page interfaces. hugetlbfs is fine, but it still requires special tools, planning, and requires some modification of the app. We can modify it with linker tricks or with LD_PRELOAD, but those certainly don't work everywhere. I was told over and over again that hugetlbfs isn't a sufficient interface for large pages, no matter how much userspace we try to stick in front of it.

Some of their apps get a 6-7x speedup from large pages!

Fragmentation also isn't an issue for a big chunk of the users since they reboot between each job.

Perhaps this policy?

In mlock(), populate huge pages if (1) the mlock range is large enough to hold some huge pages; (2) there are more than enough free high order pages.

Based on Dave's descriptions that HPC apps typically
- do mlock(), to pre-populate memory and pin them in memory
- run at fresh boot, with loads of high order pages available

Thanks,
Fengguang

Quoted 70 lines
Quoted 8 lines
Quoted 1 line

nodes going down due to memory exhaustion

Virtually all the apps in an HPC environment start up try to use all the memory they can get their hands on. With strict overcommit on, that probably means brk() or mmap() until they fail. They also usually mlock() anything they're able to allocate. Swapping is the devil to them. :)

Quoted 7 lines

Basically, what all the apps do is a recipe for stressing the VM and triggering the OOM killer. Most of the users simply hack the kernel and replace the OOM killer with one that fits their needs. Some have an attitude that "the user's app should never die" and others "the user caused this, so kill their app". Basically, there's no way to make everyone happy since they have conflicting requirements. But, this is true of the kernel in general... nothing special here.

OOM killer has been a hot topic. Have you seen Dan Malek's patches at
http://lkml.org/lkml/2009/4/13/276.

Quoted 5 lines

The split LRU should help things. It will at least make our memory scanning more efficient and ensure we're making more efficient reclaim progress. I'm not sure that anyone there knew about the oom_adjust and oom_score knobs in /proc. They do now. :)

:-)

Quoted 6 lines

One of my suggestions was to use the memory resource controller. They could give each app 95% (or whatever) of the system. This should let them keep their current "consume all memory" behavior, but stop them at sane limits.

Soft limits should help as well, basically we are trying to allow unrestricted memory access until there is contention. The patches are still under development.

Quoted 16 lines

That leads into another issue, which is the "wedding cake" software stack. There are a lot of software dependencies both in and out of the kernel. It is hard to change individual components, especially in the lower levels. This leads many of the users to use old (think 2.6.9) kernels. Nobody runs mainline, of course.

Then, there's Lustre. Everybody uses it, it's definitely a big hunk of the "wedding cake". I haven't seen any LKML postings on it in years and I really wonder how it interacts with the VM. No idea.

There's a "Hyperion cluster" which is for testing new HPC software on a decently sized cluster. One suggestion of ours was to try and get mainline tested on this every so often to look for regressions since we're not able to glean feedback from 2.6.9 kernel users. We'll see where that goes.

Quoted 6 lines
Quoted 1 line

checkpoint/restart

Many of the MPI implementations have mechanisms in userspace for checkpointing of user jobs. Most cluster administrators instruct their users to use these mechanisms. Some do. Most don't.

Good inputs and summary. Thanks!

Signature · 6 lines
	Balbir
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/
#4 lore raw
Quoted 3 lines · Wu Fengguang

Based on Dave's descriptions that HPC apps typically
- do mlock(), to pre-populate memory and pin them in memory
- run at fresh boot, with loads of high order pages available

There are definitely some of them that do this, but it certainly isn't all. It may not even be the norm.

-- Dave

#5 lore raw
Quoted 7 lines · Dave Hansen
Quoted 3 lines · Wu Fengguang

Based on Dave's descriptions that HPC apps typically
- do mlock(), to pre-populate memory and pin them in memory
- run at fresh boot, with loads of high order pages available

There are definitely some of them that do this, but it certainly isn't all. It may not even be the norm.

Some of the machine have so much memory available that 2M allocations are likely to succeed. If a machine has a couple of terabytes of memory available then its highly unlikely that a 2M allocation will not succeed.

#6 lore raw
Quoted 7 lines · Christoph Lameter
Quoted 5 lines · Dave Hansen
Quoted 3 lines · Wu Fengguang

Based on Dave's descriptions that HPC apps typically
- do mlock(), to pre-populate memory and pin them in memory
- run at fresh boot, with loads of high order pages available

Quoted 6 lines
Quoted 2 lines

There are definitely some of them that do this, but it certainly isn't all. It may not even be the norm.

Some of the machine have so much memory available that 2M allocations are likely to succeed. If a machine has a couple of terabytes of memory available then its highly unlikely that a 2M allocation will not succeed.

Huge pages used to be explicitly managed as scarce resources. As time goes by I'd imagine the move to "huge page as an optimization" POV. Optimizations that can benefit unmodified applications and can fail gracefully.