mm, meminit: Serially initialise deferred memory if trace_buf_size is specified

Yasuaki Ishimatsu reported a premature OOM when trace_buf_size=100m was specified on a machine with many CPUs. The kernel tried to allocate 38.4GB but only 16GB was available due to deferred memory initialisation.

The allocation context is within smp_init() so there are no opportunities to do the deferred meminit earlier. Furthermore, the partial initialisation of memory occurs before the size of the trace buffers is set so there is no opportunity to adjust the amount of memory that is pre-initialised. We could potentially catch when memory is low during system boot and adjust the amount that is initialised serially but it's a little clumsy as it would require a check in the failure path of the page allocator. Given that deferred meminit is basically a minor optimisation that only benefits very large machines and trace_buf_size is somewhat specialised, it follows that the most straight-forward option is to go back to serialised meminit if trace_buf_size is specified.

Reported-and-tested-by YASUAKI ISHIMATSU <[email protected]>

Signed-off-by Mel Gorman <[email protected]>

Quoted 4 lines · Mel Gorman

Yasuaki Ishimatsu reported a premature OOM when trace_buf_size=100m was specified on a machine with many CPUs. The kernel tried to allocate 38.4GB but only 16GB was available due to deferred memory initialisation.

Quoted 11 lines

The allocation context is within smp_init() so there are no opportunities to do the deferred meminit earlier. Furthermore, the partial initialisation of memory occurs before the size of the trace buffers is set so there is no opportunity to adjust the amount of memory that is pre-initialised. We could potentially catch when memory is low during system boot and adjust the amount that is initialised serially but it's a little clumsy as it would require a check in the failure path of the page allocator. Given that deferred meminit is basically a minor optimisation that only benefits very large machines and trace_buf_size is somewhat specialised, it follows that the most straight-forward option is to go back to serialised meminit if trace_buf_size is specified.

Can we instead do a smaller trace buffer in the early stage and then allocate the rest after the whole memory is initialized?

The early
memory init code is quite complex to make it even more so for something that looks like a borderline useful usecase.

Seriously, who is going
need 100M trace buffer _per cpu_ during early boot?

I doubt anyone well. Even the original reporter appeared to pick that particular value just to trigger the OOM.

On Wed 15-11-17 14:13:29, Mel Gorman wrote:
[...]

Quoted 2 lines

I doubt anyone well. Even the original reporter appeared to pick that particular value just to trigger the OOM.

Then why do we care at all? The trace buffer size can be configured from the userspace if it is not sufficiently large IIRC.

Quoted 8 lines · Michal Hocko

On Wed 15-11-17 14:13:29, Mel Gorman wrote:
[...]

Quoted 2 lines

I doubt anyone well. Even the original reporter appeared to pick that particular value just to trigger the OOM.

Then why do we care at all? The trace buffer size can be configured from the userspace if it is not sufficiently large IIRC.

I guess there is the potential that the trace buffer needs to be large enough early on in boot but I'm not sure why it would need to be that large to be honest. Bottom line, it's fairly trivial to just serialise meminit in the event that it's resized from command line. I'm also ok with just leaving this is as a "don't set the buffer that large"

Quoted 10 lines · Mel Gorman

Quoted 8 lines · Michal Hocko

On Wed 15-11-17 14:13:29, Mel Gorman wrote:
[...]

Quoted 2 lines

I doubt anyone well. Even the original reporter appeared to pick that particular value just to trigger the OOM.

Then why do we care at all? The trace buffer size can be configured from the userspace if it is not sufficiently large IIRC.

Quoted 5 lines

I guess there is the potential that the trace buffer needs to be large enough early on in boot but I'm not sure why it would need to be that large to be honest. Bottom line, it's fairly trivial to just serialise meminit in the event that it's resized from command line. I'm also ok with just leaving this is as a "don't set the buffer that large"

I would be reluctant to touch the code just because of insane kernel command line option.

That being said, I will not object or block the patch it just seems unnecessary for most reasonable setups I can think of. If there is a legitimate usage of such a large trace buffer then I wouldn't oppose.

Hi Michal and Mel,

To reproduce the issue, I specified the large trace buffer. The issue also occurs with trace_buf_size=12M and movable_node on 4.14.0.

Quoted 4 lines · YASUAKI ISHIMATSU

Hi Michal and Mel,

To reproduce the issue, I specified the large trace buffer. The issue also occurs with trace_buf_size=12M and movable_node on 4.14.0.

This is still 10x more than the default. Why do you need it in the first place? You can of course find a size that will not fit into the initial memory but I am questioning why do you want something like that during early boot in the first place.

The whole deferred struct page allocation operates under assumption that there are no large page allocator consumers that early during the boot process.

If this assumption is not correct then we probably
need a generic way to describe this. Add-hoc trace specific thing is far from idea, imho. If anything the first approach to disable the deferred initialization via kernel command line option sounds much more appropriate and simpler to me.

4. Put a check into the page allocator slowpath that triggers serialised init if the system is booting and an allocation is about to fail. It would be such a cold path that it would never be noticable although it would leave dead code in the kernel image once boot had completed

Quoted 4 lines · Mel Gorman

4. Put a check into the page allocator slowpath that triggers serialised init if the system is booting and an allocation is about to fail. It would be such a cold path that it would never be noticable although it would leave dead code in the kernel image once boot had completed

Hi Mel,

The forth approach is the best as it is seamless for admins and engineers, it will also work on any system configuration with any parameters without any special involvement.

This approach will also address the following problem:
reset_deferred_meminit() has some assumptions about how much memory we will need beforehand may break periodically as kernel requirements change. For, instance, I recently reduced amount of memory system hash tables take on large machines [1], so the comment in that function is already outdated:
/*
* Initialise at least 2G of a node but also take into account that
* two large system hashes that can take up 1GB for 0.25TB/node. */

With this approach we could always init a very small amount of struct pages, and allow the rest to be initialized on demand as boot requires until deferred struct pages are initialized. Since, having deferred pages feature assumes that the machine is large, there is no drawback of having some extra byte of dead code, especially that all the checks can be permanently switched of via static branches once deferred init is complete.

The second benefit that this approach may bring is the following: it may enable to add a new feature which would initialize struct pages on demand later, when needed by applications. This feature would be configurable or enabled via kernel parameter (not sure which is better).

if (allocation is failing)
if (uninit struct pages available)
init enought to finish alloc

Again, once all pages are initialized, the checks will be turned off via static branching, so I think the code can be shared.

Here is the rationale for this feature:

Each physical machine may run a very large number of linux containers. Steve Sistare (CCed), recently studied how much memory each instance of clear container is taking, and it turns out to be about 125 MB, when containers are booted with 2G of memory and 1 CPU. Out of those 125 MB, 32 MB is consumed by struct page array as we use 64-bytes per page. Admins tend to be protective in the amount of memory that is configured, therefore they may over-commit the amount of memory that is actually required by the container. So, by allowing struct pages to be initialized only on demand, we can save around 25% of the memory that is consumed by fresh instance of container. Now, that struct pages are not zeroed during boot [2], and if we will implement the forth option, we can get closer to implementing a complete on demand struct page initialization.

I can volunteer to work on these projects.

A lack of involvement from admins is indeed desirable. For example, while I might concede on using a disable-everything-switch, I would not be happy to introduce a switch that specified how much memory per node to initialise.

For the forth approach, I really would be only thinking of a blunt "initialise everything instead of going OOM". I was wary of making things too complicated and I worried about some side-effects I'll cover later.

Quoted 8 lines

With this approach we could always init a very small amount of struct pages, and allow the rest to be initialized on demand as boot requires until deferred struct pages are initialized. Since, having deferred pages feature assumes that the machine is large, there is no drawback of having some extra byte of dead code, especially that all the checks can be permanently switched of via static branches once deferred init is complete.

This is where I fear there may be dragons. If we minimse the number of struct pages and initialise serially as necessary, there is a danger that we'll allocate remote memory in cases where local memory would have done because a remote node had enough memory.

necessary at boot-time to force allocations from local node where possible and initialise more memory as necessary. That starts getting complicated because we'd need to adjust gfp-flags in the fast path with init-and-retry logic in the slow path and that could be a constant penalty. We could offset that in the fast path by using static branches

but it's getting more and
more complex for what is a minor optimisation -- shorter boot times on large machines where userspace itself could take a *long* time to get up and running (think database reading in 1TB of data from disk as it warms up).

Quoted 11 lines

The second benefit that this approach may bring is the following: it may enable to add a new feature which would initialize struct pages on demand later, when needed by applications. This feature would be configurable or enabled via kernel parameter (not sure which is better).

if (allocation is failing)
if (uninit struct pages available)
init enought to finish alloc

There is a hidden hazard with this as well -- benchmarking. Early in the lifetime of the system, it's going to be slower because we're initialising memory while measuring performance when previously no such work would be necessary. While it's somewhat of a corner-case, it's still real and it would generate reports. For example, I got burned once by a "regression" that was due to ext4's lazy_init feature because IO benchmarks appeared slower when in reality, it was only due to a fresh filesystem initialising. It was necessary to turn off the feature at mkfs time to get accurate measurements. I think the same could happen with memory and we'd have to special case some things.

We'd want to be *very* sure there was a substantial benefit to the complexity. For benchmarking a system, we'd also need to be able to turn it off.

Quoted 19 lines

Again, once all pages are initialized, the checks will be turned off via static branching, so I think the code can be shared.

Here is the rationale for this feature:

Each physical machine may run a very large number of linux containers. Steve Sistare (CCed), recently studied how much memory each instance of clear container is taking, and it turns out to be about 125 MB, when containers are booted with 2G of memory and 1 CPU. Out of those 125 MB, 32 MB is consumed by struct page array as we use 64-bytes per page. Admins tend to be protective in the amount of memory that is configured, therefore they may over-commit the amount of memory that is actually required by the container. So, by allowing struct pages to be initialized only on demand, we can save around 25% of the memory that is consumed by fresh instance of container. Now, that struct pages are not zeroed during boot [2], and if we will implement the forth option, we can get closer to implementing a complete on demand struct page initialization.

Quoted 2 lines

I can volunteer to work on these projects.

I accept the potential for packing more containers into the system but While I commend the work you've done so far, I'd be wary and warn you of going too far down this path. I wouldn't NAK patches going in this direction as long as they would eventually be behind static branches but I don't feel it's the most urgent problem to work on either. This is why, even if I took the fourth option, that it would be a blunt "init everything if we're going OOM" approach. However, that is my opinion and it's partially based on a lack of sensible use cases. I suspect you have better justification that would be included in changelogs.

Hi Mel,

Thank you very much for your feedback, my replies below:

Quoted 8 lines

A lack of involvement from admins is indeed desirable. For example, while I might concede on using a disable-everything-switch, I would not be happy to introduce a switch that specified how much memory per node to initialise.

For the forth approach, I really would be only thinking of a blunt "initialise everything instead of going OOM". I was wary of making things too complicated and I worried about some side-effects I'll cover later.

I see, I misunderstood your suggestion. Switching to serial initialization when OOM works, however, boot time becomes unpredictable, with some configurations boot is fast with others it is slow. All of that depends on whether predictions in reset_deferred_meminit() were good or not which is not easy to debug for users. Also, overtime predictions in reset_deferred_meminit() can become very off, and I do not think that we want to continuously adjust this function.

Quoted 9 lines

Quoted 8 lines

With this approach we could always init a very small amount of struct pages, and allow the rest to be initialized on demand as boot requires until deferred struct pages are initialized. Since, having deferred pages feature assumes that the machine is large, there is no drawback of having some extra byte of dead code, especially that all the checks can be permanently switched of via static branches once deferred init is complete.

Quoted 4 lines

This is where I fear there may be dragons. If we minimse the number of struct pages and initialise serially as necessary, there is a danger that we'll allocate remote memory in cases where local memory would have done because a remote node had enough memory.

True, but is not what we have now has the same issue as well? If one node is gets out of memory we start using memory from another node, before deferred pages are initialized?

To offset that risk, it would be

Quoted 5 lines

necessary at boot-time to force allocations from local node where possible and initialise more memory as necessary. That starts getting complicated because we'd need to adjust gfp-flags in the fast path with init-and-retry logic in the slow path and that could be a constant penalty. We could offset that in the fast path by using static branches

I will try to implement this, and see how complicated the patch will be, if it gets too complicated for the problem I am trying to solve we can return to one of your suggestions.

I was thinking to do something like this:

Start with every small amount of initialized pages in every node. If allocation fails, initialize enough struct pages to cover this particular allocation with struct pages rounded up to section size but in every single node.

Quoted 4 lines

but it's getting more and
more complex for what is a minor optimisation -- shorter boot times on large machines where userspace itself could take a *long* time to get up and running (think database reading in 1TB of data from disk as it warms up).

On M6-32 with 32T [1] of memory it saves over 4 minutes of boot time, and this is on SPARC with 8K pages, on x86 it would be around of 8 minutes because of twice as many pages. This feature improves availability for larger machines quite a bit. Overtime, systems are growing, so I expect this feature to become a default configuration in the next several years on server configs.

Quoted 97 lines

-----Original Message-----
From: Mel Gorman [mailto:[email protected]]
Sent: Wednesday, December 06, 2017 5:50 AM
To: Pavel Tatashin <[email protected]>
Cc: Michal Hocko <[email protected]>; YASUAKI ISHIMATSU <[email protected]>; Andrew Morton <[email protected]>; Linux Memory Management List <[email protected]>; linux- [email protected]; Sanagi, Koki <[email protected]>; Steve Sistare <[email protected]>
Subject: Re: [PATCH] mm, meminit: Serially initialise deferred memory if trace_buf_size is specified

Quoted 13 lines · Pavel Tatashin

Hi Mel,

Thank you very much for your feedback, my replies below:

Quoted 8 lines

A lack of involvement from admins is indeed desirable. For example, while I might concede on using a disable-everything-switch, I would not be happy to introduce a switch that specified how much memory per node to initialise.

For the forth approach, I really would be only thinking of a blunt "initialise everything instead of going OOM". I was wary of making things too complicated and I worried about some side-effects I'll cover later.

Quoted 9 lines

I see, I misunderstood your suggestion. Switching to serial initialization when OOM works, however, boot time becomes unpredictable, with some configurations boot is fast with others it is slow. All of that depends on whether predictions in reset_deferred_meminit() were good or not which is not easy to debug for users. Also, overtime predictions in reset_deferred_meminit() can become very off, and I do not think that we want to continuously adjust this function.

You could increase the probabilty of a report by doing a WARN_ON_ONCE if the serialised meminit is used.

Quoted 9 lines

Quoted 9 lines

Quoted 8 lines

With this approach we could always init a very small amount of struct pages, and allow the rest to be initialized on demand as boot requires until deferred struct pages are initialized. Since, having deferred pages feature assumes that the machine is large, there is no drawback of having some extra byte of dead code, especially that all the checks can be permanently switched of via static branches once deferred init is complete.

Quoted 9 lines

Quoted 4 lines

This is where I fear there may be dragons. If we minimse the number of struct pages and initialise serially as necessary, there is a danger that we'll allocate remote memory in cases where local memory would have done because a remote node had enough memory.

True, but is not what we have now has the same issue as well? If one node is gets out of memory we start using memory from another node, before deferred pages are initialized?

It's possible but I'm not aware of it happening currently.

Quoted 12 lines

To offset that risk, it would be

Quoted 6 lines

necessary at boot-time to force allocations from local node where possible and initialise more memory as necessary. That starts getting complicated because we'd need to adjust gfp-flags in the fast path with init-and-retry logic in the slow path and that could be a constant penalty. We could offset that in the fast path by using static branches

I will try to implement this, and see how complicated the patch will be, if it gets too complicated for the problem I am trying to solve we can return to one of your suggestions.

Quoted 7 lines

I was thinking to do something like this:

Start with every small amount of initialized pages in every node. If allocation fails, initialize enough struct pages to cover this particular allocation with struct pages rounded up to section size but in every single node.

Ok, just make sure it's all in the slow paths of the allocator when the alternative is to fail the allocation.

Quoted 4 lines

Quoted 4 lines

but it's getting more and
more complex for what is a minor optimisation -- shorter boot times on large machines where userspace itself could take a *long* time to get up and running (think database reading in 1TB of data from disk as it

warms up).

Quoted 8 lines

On M6-32 with 32T [1] of memory it saves over 4 minutes of boot time, and this is on SPARC with 8K pages, on x86 it would be around of 8 minutes because of twice as many pages. This feature improves availability for larger machines quite a bit. Overtime, systems are growing, so I expect this feature to become a default configuration in the next several years on server configs.

Quoted 6 lines

Ok, when developing the series originally, I had no machine even close to 32T of memory.

--
Mel Gorman
SUSE Labs

Pavel,

I assume you are working on the fix.
Do you have any progress ?

Koki

Quoted 97 lines

Quoted 97 lines

-----Original Message-----
From: Mel Gorman [mailto:[email protected]]
Sent: Wednesday, December 06, 2017 5:50 AM
To: Pavel Tatashin <[email protected]>
Cc: Michal Hocko <[email protected]>; YASUAKI ISHIMATSU <[email protected]>; Andrew Morton <[email protected]>; Linux Memory Management List <[email protected]>; linux- [email protected]; Sanagi, Koki <[email protected]>; Steve Sistare <[email protected]>
Subject: Re: [PATCH] mm, meminit: Serially initialise deferred memory if trace_buf_size is specified

Quoted 13 lines · Pavel Tatashin

Hi Mel,

Thank you very much for your feedback, my replies below:

Quoted 8 lines

A lack of involvement from admins is indeed desirable. For example, while I might concede on using a disable-everything-switch, I would not be happy to introduce a switch that specified how much memory per node to initialise.

For the forth approach, I really would be only thinking of a blunt "initialise everything instead of going OOM". I was wary of making things too complicated and I worried about some side-effects I'll cover later.

Quoted 9 lines

I see, I misunderstood your suggestion. Switching to serial initialization when OOM works, however, boot time becomes unpredictable, with some configurations boot is fast with others it is slow. All of that depends on whether predictions in reset_deferred_meminit() were good or not which is not easy to debug for users. Also, overtime predictions in reset_deferred_meminit() can become very off, and I do not think that we want to continuously adjust this function.

You could increase the probabilty of a report by doing a WARN_ON_ONCE if the serialised meminit is used.

Quoted 9 lines

Quoted 9 lines

Quoted 8 lines

With this approach we could always init a very small amount of struct pages, and allow the rest to be initialized on demand as boot requires until deferred struct pages are initialized. Since, having deferred pages feature assumes that the machine is large, there is no drawback of having some extra byte of dead code, especially that all the checks can be permanently switched of via static branches once deferred init is complete.

Quoted 9 lines

Quoted 4 lines

This is where I fear there may be dragons. If we minimse the number of struct pages and initialise serially as necessary, there is a danger that we'll allocate remote memory in cases where local memory would have done because a remote node had enough memory.

True, but is not what we have now has the same issue as well? If one node is gets out of memory we start using memory from another node, before deferred pages are initialized?

It's possible but I'm not aware of it happening currently.

Quoted 12 lines

To offset that risk, it would be

Quoted 6 lines

necessary at boot-time to force allocations from local node where possible and initialise more memory as necessary. That starts getting complicated because we'd need to adjust gfp-flags in the fast path with init-and-retry logic in the slow path and that could be a constant penalty. We could offset that in the fast path by using static branches

I will try to implement this, and see how complicated the patch will be, if it gets too complicated for the problem I am trying to solve we can return to one of your suggestions.

Quoted 7 lines

I was thinking to do something like this:

Start with every small amount of initialized pages in every node. If allocation fails, initialize enough struct pages to cover this particular allocation with struct pages rounded up to section size but in every single node.

Ok, just make sure it's all in the slow paths of the allocator when the alternative is to fail the allocation.

Quoted 4 lines

Quoted 4 lines

but it's getting more and
more complex for what is a minor optimisation -- shorter boot times on large machines where userspace itself could take a *long* time to get up and running (think database reading in 1TB of data from disk as it

warms up).

Quoted 8 lines

On M6-32 with 32T [1] of memory it saves over 4 minutes of boot time, and this is on SPARC with 8K pages, on x86 it would be around of 8 minutes because of twice as many pages. This feature improves availability for larger machines quite a bit. Overtime, systems are growing, so I expect this feature to become a default configuration in the next several years on server configs.

Quoted 6 lines

Quoted 6 lines

Ok, when developing the series originally, I had no machine even close to 32T of memory.

--
Mel Gorman
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a hrefmailto:"[email protected]"> [email protected] </a>

Yes, the patch is here:
https://lkml.org/lkml/2018/1/12/600

Hi Koki,

Yes, the patch is here:
https://lkml.org/lkml/2018/1/12/600

It has not been reviewed yet.

Pavel

Quoted 116 lines · On Wed, Jan 31, 2018 at 12:28 PM, [email protected]…

Pavel,

I assume you are working on the fix.
Do you have any progress ?

Koki

Quoted 97 lines

Quoted 97 lines

-----Original Message-----
From: Mel Gorman [mailto:[email protected]]
Sent: Wednesday, December 06, 2017 5:50 AM
To: Pavel Tatashin <[email protected]>
Cc: Michal Hocko <[email protected]>; YASUAKI ISHIMATSU <[email protected]>; Andrew Morton <[email protected]>; Linux Memory Management List <[email protected]>; linux- [email protected]; Sanagi, Koki <[email protected]>; Steve Sistare <[email protected]>
Subject: Re: [PATCH] mm, meminit: Serially initialise deferred memory if trace_buf_size is specified

Quoted 13 lines · Pavel Tatashin

Hi Mel,

Thank you very much for your feedback, my replies below:

Quoted 8 lines

A lack of involvement from admins is indeed desirable. For example, while I might concede on using a disable-everything-switch, I would not be happy to introduce a switch that specified how much memory per node to initialise.

For the forth approach, I really would be only thinking of a blunt "initialise everything instead of going OOM". I was wary of making things too complicated and I worried about some side-effects I'll cover later.

Quoted 9 lines

I see, I misunderstood your suggestion. Switching to serial initialization when OOM works, however, boot time becomes unpredictable, with some configurations boot is fast with others it is slow. All of that depends on whether predictions in reset_deferred_meminit() were good or not which is not easy to debug for users. Also, overtime predictions in reset_deferred_meminit() can become very off, and I do not think that we want to continuously adjust this function.

You could increase the probabilty of a report by doing a WARN_ON_ONCE if the serialised meminit is used.

Quoted 9 lines

Quoted 9 lines

Quoted 8 lines

With this approach we could always init a very small amount of struct pages, and allow the rest to be initialized on demand as boot requires until deferred struct pages are initialized. Since, having deferred pages feature assumes that the machine is large, there is no drawback of having some extra byte of dead code, especially that all the checks can be permanently switched of via static branches once deferred init is complete.

Quoted 9 lines

Quoted 4 lines

This is where I fear there may be dragons. If we minimse the number of struct pages and initialise serially as necessary, there is a danger that we'll allocate remote memory in cases where local memory would have done because a remote node had enough memory.

True, but is not what we have now has the same issue as well? If one node is gets out of memory we start using memory from another node, before deferred pages are initialized?

It's possible but I'm not aware of it happening currently.

Quoted 12 lines

To offset that risk, it would be

Quoted 6 lines

necessary at boot-time to force allocations from local node where possible and initialise more memory as necessary. That starts getting complicated because we'd need to adjust gfp-flags in the fast path with init-and-retry logic in the slow path and that could be a constant penalty. We could offset that in the fast path by using static branches

I will try to implement this, and see how complicated the patch will be, if it gets too complicated for the problem I am trying to solve we can return to one of your suggestions.

Quoted 7 lines

I was thinking to do something like this:

Start with every small amount of initialized pages in every node. If allocation fails, initialize enough struct pages to cover this particular allocation with struct pages rounded up to section size but in every single node.

Ok, just make sure it's all in the slow paths of the allocator when the alternative is to fail the allocation.

Quoted 4 lines

Quoted 4 lines

but it's getting more and
more complex for what is a minor optimisation -- shorter boot times on large machines where userspace itself could take a *long* time to get up and running (think database reading in 1TB of data from disk as it

warms up).

Quoted 8 lines

On M6-32 with 32T [1] of memory it saves over 4 minutes of boot time, and this is on SPARC with 8K pages, on x86 it would be around of 8 minutes because of twice as many pages. This feature improves availability for larger machines quite a bit. Overtime, systems are growing, so I expect this feature to become a default configuration in the next several years on server configs.

Quoted 6 lines

Quoted 6 lines

Ok, when developing the series originally, I had no machine even close to 32T of memory.

--
Mel Gorman
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a hrefmailto:"[email protected]"> [email protected] </a>

Hello Pavel,

Quoted 2 lines

Yes, the patch is here:
https://lkml.org/lkml/2018/1/12/600

I tested your patch in my box and it worked well. Please feel free to add the following.

Tested-by Masayoshi Mizuma <[email protected]>

You may repost the patch after adding your reply for Andrew's comment as [PATCH 0/1]...

- Masayoshi

Quoted 5 lines · Wed, 31 Jan 2018 13:24:55 -0500 Pavel Tatashin wrote:

Hi Koki,

Yes, the patch is here:
https://lkml.org/lkml/2018/1/12/600

Quoted 122 lines

It has not been reviewed yet.

Pavel

Quoted 116 lines · On Wed, Jan 31, 2018 at 12:28 PM, [email protected]…

Pavel,

I assume you are working on the fix.
Do you have any progress ?

Koki

Quoted 97 lines

Quoted 97 lines

-----Original Message-----
From: Mel Gorman [mailto:[email protected]]
Sent: Wednesday, December 06, 2017 5:50 AM
To: Pavel Tatashin <[email protected]>
Cc: Michal Hocko <[email protected]>; YASUAKI ISHIMATSU <[email protected]>; Andrew Morton <[email protected]>; Linux Memory Management List <[email protected]>; linux- [email protected]; Sanagi, Koki <[email protected]>; Steve Sistare <[email protected]>
Subject: Re: [PATCH] mm, meminit: Serially initialise deferred memory if trace_buf_size is specified

Quoted 13 lines · Pavel Tatashin

Hi Mel,

Thank you very much for your feedback, my replies below:

Quoted 8 lines

A lack of involvement from admins is indeed desirable. For example, while I might concede on using a disable-everything-switch, I would not be happy to introduce a switch that specified how much memory per node to initialise.

For the forth approach, I really would be only thinking of a blunt "initialise everything instead of going OOM". I was wary of making things too complicated and I worried about some side-effects I'll cover later.

Quoted 9 lines

I see, I misunderstood your suggestion. Switching to serial initialization when OOM works, however, boot time becomes unpredictable, with some configurations boot is fast with others it is slow. All of that depends on whether predictions in reset_deferred_meminit() were good or not which is not easy to debug for users. Also, overtime predictions in reset_deferred_meminit() can become very off, and I do not think that we want to continuously adjust this function.

You could increase the probabilty of a report by doing a WARN_ON_ONCE if the serialised meminit is used.

Quoted 9 lines

Quoted 9 lines

Quoted 8 lines

With this approach we could always init a very small amount of struct pages, and allow the rest to be initialized on demand as boot requires until deferred struct pages are initialized. Since, having deferred pages feature assumes that the machine is large, there is no drawback of having some extra byte of dead code, especially that all the checks can be permanently switched of via static branches once deferred init is complete.

Quoted 9 lines

Quoted 4 lines

This is where I fear there may be dragons. If we minimse the number of struct pages and initialise serially as necessary, there is a danger that we'll allocate remote memory in cases where local memory would have done because a remote node had enough memory.

True, but is not what we have now has the same issue as well? If one node is gets out of memory we start using memory from another node, before deferred pages are initialized?

It's possible but I'm not aware of it happening currently.

Quoted 12 lines

To offset that risk, it would be

Quoted 6 lines

necessary at boot-time to force allocations from local node where possible and initialise more memory as necessary. That starts getting complicated because we'd need to adjust gfp-flags in the fast path with init-and-retry logic in the slow path and that could be a constant penalty. We could offset that in the fast path by using static branches

I will try to implement this, and see how complicated the patch will be, if it gets too complicated for the problem I am trying to solve we can return to one of your suggestions.

Quoted 7 lines

I was thinking to do something like this:

Start with every small amount of initialized pages in every node. If allocation fails, initialize enough struct pages to cover this particular allocation with struct pages rounded up to section size but in every single node.

Ok, just make sure it's all in the slow paths of the allocator when the alternative is to fail the allocation.

Quoted 4 lines

Quoted 4 lines

but it's getting more and
more complex for what is a minor optimisation -- shorter boot times on large machines where userspace itself could take a *long* time to get up and running (think database reading in 1TB of data from disk as it

warms up).

Quoted 8 lines

On M6-32 with 32T [1] of memory it saves over 4 minutes of boot time, and this is on SPARC with 8K pages, on x86 it would be around of 8 minutes because of twice as many pages. This feature improves availability for larger machines quite a bit. Overtime, systems are growing, so I expect this feature to become a default configuration in the next several years on server configs.

Quoted 6 lines

Quoted 6 lines

Ok, when developing the series originally, I had no machine even close to 32T of memory.

--
Mel Gorman
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a hrefmailto:"[email protected]"> [email protected] </a>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Quoted 4 lines · Mel Gorman

4. Put a check into the page allocator slowpath that triggers serialised init if the system is booting and an allocation is about to fail. It would be such a cold path that it would never be noticable although it would leave dead code in the kernel image once boot had completed

Hi Mel,

The forth approach is the best as it is seamless for admins and engineers, it will also work on any system configuration with any parameters without any special involvement.

Quoted 5 lines

Hi Mel,

The forth approach is the best as it is seamless for admins and engineers, it will also work on any system configuration with any parameters without any special involvement.

Apart from what-mel-said, I'd be concerned that this failsafe would almost never get tested. We should find some way to ensure that this code gets exercised in some people's kernels on a permanent basis and I'm not sure how to do that.

One option might be to ask Fengguang to add the occasional test_pavels_stuff=1 to the kernel boot commandline. That's better than nothing but 0-day only runs on a small number of machine types.

Yasuaki Ishimatsu reported a premature OOM when trace_buf_size=100m was specified on a machine with many CPUs. The kernel tried to allocate 38.4GB but only 16GB was available due to deferred memory initialisation.

The allocation context is within smp_init() so there are no opportunities to do the deferred meminit earlier. Furthermore, the partial initialisation of memory occurs before the size of the trace buffers is set so there is no opportunity to adjust the amount of memory that is pre-initialised. We could potentially catch when memory is low during system boot and adjust the amount that is initialised serially but it's a little clumsy as it would require a check in the failure path of the page allocator. Given that deferred meminit is basically a minor optimisation that only benefits very large machines and trace_buf_size is somewhat specialised, it follows that the most straight-forward option is to go back to serialised meminit if trace_buf_size is specified.

Quoted 4 lines · On Wed, 15 Nov 2017 08:55:56 +0000 Mel Gorman <[email protected]>…

Yasuaki Ishimatsu reported a premature OOM when trace_buf_size=100m was specified on a machine with many CPUs. The kernel tried to allocate 38.4GB but only 16GB was available due to deferred memory initialisation.

Quoted 11 lines

The allocation context is within smp_init() so there are no opportunities to do the deferred meminit earlier. Furthermore, the partial initialisation of memory occurs before the size of the trace buffers is set so there is no opportunity to adjust the amount of memory that is pre-initialised. We could potentially catch when memory is low during system boot and adjust the amount that is initialised serially but it's a little clumsy as it would require a check in the failure path of the page allocator. Given that deferred meminit is basically a minor optimisation that only benefits very large machines and trace_buf_size is somewhat specialised, it follows that the most straight-forward option is to go back to serialised meminit if trace_buf_size is specified.

Patch is rather messy.

I went cross-eyed trying to work out how tracing allocates that buffer, but I assume it ends up somewhere in the page allocator.

If the page
allocator is about to fail an allocation request and sees that memory initialization is still ongoing, surely the page allocator should just wait? That seems to be the most general fix?