Skip to contentSkip to conversation

Tiered memory accounting and management

LSF/MM TOPIC Tim Chen 3 participants 9 messages Jun 14, 2021 – Jun 21, 2021 Memory management

lore
Share & tools
lore

Tim Chen

author
#0 lore raw

From: Tim Chen <[email protected]>

Tiered memory accounting and management
------------------------------------------------------------
Traditionally, all RAM is DRAM. Some DRAM might be closer/faster than others, but a byte of media has about the same cost whether it is close or far. But, with new memory tiers such as High-Bandwidth Memory or Persistent Memory, there is a choice between fast/expensive and slow/cheap. But, the current memory cgroups still live in the old model. There is only one set of limits, and it implies that all memory has the same cost. We would like to extend memory cgroups to comprehend different memory tiers to give users a way to choose a mix between fast/expensive and slow/cheap.

To manage such memory, we will need to account memory usage and impose limits for each kind of memory.

There were a couple of approaches that have been discussed previously to partition the memory between the cgroups listed below. We will like to
use the LSF/MM session to come to a consensus on the approach to take.

1. Per NUMA node limit and accounting for each cgroup.
We can assign higher limits on better performing memory node for higher priority cgroups.

There are some loose ends here that warrant further discussions:
(1) A user friendly interface for such limits. Will a proportional weight for the cgroup that translate to actual absolute limit be more suitable?
(2) Memory mis-configurations can occur more easily as the admin has a much larger number of limits spread among between the cgroups to manage. Over-restrictive limits can lead to under utilized and wasted memory and hurt performance.
(3) OOM behavior when a cgroup hits its limit.

2. Per memory tier limit and accounting for each cgroup.
We can assign higher limits on memories in better performing memory tier for higher priority cgroups. I previously prototyped a soft limit based implementation to demonstrate the tiered limit idea.

There are also a number of issues here:
(1) The advantage is we have fewer limits to deal with simplifying
configuration. However, there are doubts raised by a number of people on whether we can really properly classify the NUMA nodes into memory tiers. There could still be significant performance differences between NUMA nodes even for the same kind of memory. We will also not have the fine-grained control and flexibility that comes with a per NUMA node limit.
(2) Will a memory hierarchy defined by promotion/demotion relationship between
memory nodes be a viable approach for defining memory tiers?

These issues related to the management of systems with multiple kind of memories can be ironed out in this session.

#1 lore raw
Quoted 53 lines · Tim Chen

From: Tim Chen <[email protected]>

Tiered memory accounting and management
------------------------------------------------------------
Traditionally, all RAM is DRAM. Some DRAM might be closer/faster than others, but a byte of media has about the same cost whether it is close or far. But, with new memory tiers such as High-Bandwidth Memory or Persistent Memory, there is a choice between fast/expensive and slow/cheap. But, the current memory cgroups still live in the old model. There is only one set of limits, and it implies that all memory has the same cost. We would like to extend memory cgroups to comprehend different memory tiers to give users a way to choose a mix between fast/expensive and slow/cheap.

To manage such memory, we will need to account memory usage and impose limits for each kind of memory.

There were a couple of approaches that have been discussed previously to partition the memory between the cgroups listed below. We will like to
use the LSF/MM session to come to a consensus on the approach to take.

1. Per NUMA node limit and accounting for each cgroup.
We can assign higher limits on better performing memory node for higher priority cgroups.

There are some loose ends here that warrant further discussions:
(1) A user friendly interface for such limits. Will a proportional weight for the cgroup that translate to actual absolute limit be more suitable?
(2) Memory mis-configurations can occur more easily as the admin has a much larger number of limits spread among between the cgroups to manage. Over-restrictive limits can lead to under utilized and wasted memory and hurt performance.
(3) OOM behavior when a cgroup hits its limit.

2. Per memory tier limit and accounting for each cgroup.
We can assign higher limits on memories in better performing memory tier for higher priority cgroups. I previously prototyped a soft limit based implementation to demonstrate the tiered limit idea.

There are also a number of issues here:
(1) The advantage is we have fewer limits to deal with simplifying
configuration. However, there are doubts raised by a number of people on whether we can really properly classify the NUMA nodes into memory tiers. There could still be significant performance differences between NUMA nodes even for the same kind of memory. We will also not have the fine-grained control and flexibility that comes with a per NUMA node limit.
(2) Will a memory hierarchy defined by promotion/demotion relationship between
memory nodes be a viable approach for defining memory tiers?

Quoted 2 lines

These issues related to the management of systems with multiple kind of memories can be ironed out in this session.

Thanks for suggesting this topic. I'm interested in the topic and would like to attend.

Other than the above points. I'm wondering whether we shall discuss "Migrate Pages in lieu of discard" as well? Dave Hansen is driving the development and I have been involved in the early development and review, but it seems there are still some open questions according to the latest review feedback.

Some other folks may be interested in this topic either, CC'ed them in the thread.

Quoted 1 line
#2 lore raw

Thanks Yang for the CC.

Quoted 29 lines · Yang Shi
Quoted 27 lines · Tim Chen

From: Tim Chen <[email protected]>

Tiered memory accounting and management
------------------------------------------------------------
Traditionally, all RAM is DRAM. Some DRAM might be closer/faster than others, but a byte of media has about the same cost whether it is close or far. But, with new memory tiers such as High-Bandwidth Memory or Persistent Memory, there is a choice between fast/expensive and slow/cheap. But, the current memory cgroups still live in the old model. There is only one set of limits, and it implies that all memory has the same cost. We would like to extend memory cgroups to comprehend different memory tiers to give users a way to choose a mix between fast/expensive and slow/cheap.

To manage such memory, we will need to account memory usage and impose limits for each kind of memory.

There were a couple of approaches that have been discussed previously to partition the memory between the cgroups listed below. We will like to
use the LSF/MM session to come to a consensus on the approach to take.

1. Per NUMA node limit and accounting for each cgroup.
We can assign higher limits on better performing memory node for higher priority cgroups.

Quoted 9 lines
Quoted 9 lines

There are some loose ends here that warrant further discussions:
(1) A user friendly interface for such limits. Will a proportional weight for the cgroup that translate to actual absolute limit be more suitable?
(2) Memory mis-configurations can occur more easily as the admin has a much larger number of limits spread among between the cgroups to manage. Over-restrictive limits can lead to under utilized and wasted memory and hurt performance.
(3) OOM behavior when a cgroup hits its limit.

This (numa based limits) is something I was pushing for but after discussing this internally with userspace controller devs, I have to backoff from this position.

The main feedback I got was that setting one memory limit is already complicated and having to set/adjust these many limits would be horrifying.

Quoted 23 lines
Quoted 17 lines

2. Per memory tier limit and accounting for each cgroup.
We can assign higher limits on memories in better performing memory tier for higher priority cgroups. I previously prototyped a soft limit based implementation to demonstrate the tiered limit idea.

There are also a number of issues here:
(1) The advantage is we have fewer limits to deal with simplifying
configuration. However, there are doubts raised by a number of people on whether we can really properly classify the NUMA nodes into memory tiers. There could still be significant performance differences between NUMA nodes even for the same kind of memory. We will also not have the fine-grained control and flexibility that comes with a per NUMA node limit.
(2) Will a memory hierarchy defined by promotion/demotion relationship between
memory nodes be a viable approach for defining memory tiers?

Quoted 2 lines

These issues related to the management of systems with multiple kind of memories can be ironed out in this session.

Thanks for suggesting this topic. I'm interested in the topic and would like to attend.

Quoted 9 lines

Other than the above points. I'm wondering whether we shall discuss "Migrate Pages in lieu of discard" as well? Dave Hansen is driving the development and I have been involved in the early development and review, but it seems there are still some open questions according to the latest review feedback.

Some other folks may be interested in this topic either, CC'ed them in the thread.

At the moment "personally" I am more inclined towards a passive approach towards the memcg accounting of memory tiers. By that I mean, let's start by providing a 'usage' interface and get more production/real-world data to motivate the 'limit' interfaces. (One minor reason is that defining the 'limit' interface will force us to make the decision on defining tiers i.e. numa or a set of numa or others).

IMHO we should focus more on the "aging" of the application memory and "migration/balance" between the tiers. I don't think the memory reclaim infrastructure is the right place for these operations (unevictable pages are ignored and not accurate ages). What we need is proactive continuous aging and balancing. We need something like, with additions, Multi-gen LRUs or DAMON or page idle tracking for aging and a new mechanism for balancing which takes ages into account.

To give a more concrete example: Let's say we have a system with two memory tiers and multiple low and high priority jobs. For high priority jobs, set the allocation try list from high to low tier and for low priority jobs the reverse of that (I am not sure if we can do that out of the box with today's kernel). In the background we migrate cold memory down the tiers and hot memory in the reverse direction.

In this background mechanism we can enforce all different limiting policies like Yang's original high and low tier percentage or something like X% of accesses of high priority jobs should be from high tier. Basically I am saying until we find from production data that this background mechanism is not strong enough to enforce passive limits, we should delay the decision on limit interfaces.

author
#3 lore raw
Quoted 83 lines · Shakeel Butt

Thanks Yang for the CC.

Quoted 29 lines · Yang Shi
Quoted 27 lines · Tim Chen

From: Tim Chen <[email protected]>

Tiered memory accounting and management
------------------------------------------------------------
Traditionally, all RAM is DRAM. Some DRAM might be closer/faster than others, but a byte of media has about the same cost whether it is close or far. But, with new memory tiers such as High-Bandwidth Memory or Persistent Memory, there is a choice between fast/expensive and slow/cheap. But, the current memory cgroups still live in the old model. There is only one set of limits, and it implies that all memory has the same cost. We would like to extend memory cgroups to comprehend different memory tiers to give users a way to choose a mix between fast/expensive and slow/cheap.

To manage such memory, we will need to account memory usage and impose limits for each kind of memory.

There were a couple of approaches that have been discussed previously to partition the memory between the cgroups listed below. We will like to
use the LSF/MM session to come to a consensus on the approach to take.

1. Per NUMA node limit and accounting for each cgroup.
We can assign higher limits on better performing memory node for higher priority cgroups.

Quoted 9 lines
Quoted 9 lines

There are some loose ends here that warrant further discussions:
(1) A user friendly interface for such limits. Will a proportional weight for the cgroup that translate to actual absolute limit be more suitable?
(2) Memory mis-configurations can occur more easily as the admin has a much larger number of limits spread among between the cgroups to manage. Over-restrictive limits can lead to under utilized and wasted memory and hurt performance.
(3) OOM behavior when a cgroup hits its limit.

This (numa based limits) is something I was pushing for but after discussing this internally with userspace controller devs, I have to backoff from this position.

The main feedback I got was that setting one memory limit is already complicated and having to set/adjust these many limits would be horrifying.

Quoted 23 lines
Quoted 17 lines

2. Per memory tier limit and accounting for each cgroup.
We can assign higher limits on memories in better performing memory tier for higher priority cgroups. I previously prototyped a soft limit based implementation to demonstrate the tiered limit idea.

There are also a number of issues here:
(1) The advantage is we have fewer limits to deal with simplifying
configuration. However, there are doubts raised by a number of people on whether we can really properly classify the NUMA nodes into memory tiers. There could still be significant performance differences between NUMA nodes even for the same kind of memory. We will also not have the fine-grained control and flexibility that comes with a per NUMA node limit.
(2) Will a memory hierarchy defined by promotion/demotion relationship between
memory nodes be a viable approach for defining memory tiers?

Quoted 2 lines

These issues related to the management of systems with multiple kind of memories can be ironed out in this session.

Thanks for suggesting this topic. I'm interested in the topic and would like to attend.

Quoted 9 lines

Other than the above points. I'm wondering whether we shall discuss "Migrate Pages in lieu of discard" as well? Dave Hansen is driving the development and I have been involved in the early development and review, but it seems there are still some open questions according to the latest review feedback.

Some other folks may be interested in this topic either, CC'ed them in the thread.

Quoted 7 lines

At the moment "personally" I am more inclined towards a passive approach towards the memcg accounting of memory tiers. By that I mean, let's start by providing a 'usage' interface and get more production/real-world data to motivate the 'limit' interfaces. (One minor reason is that defining the 'limit' interface will force us to make the decision on defining tiers i.e. numa or a set of numa or others).

Probably we could first start with accounting the memory used in each NUMA node for a cgroup and exposing this information to user space. I think that is useful regardless.

There is still a question of whether we want to define a set of numa node or tier and extend the accounting and management at that memory tier abstraction level.

Quoted 8 lines

IMHO we should focus more on the "aging" of the application memory and "migration/balance" between the tiers. I don't think the memory reclaim infrastructure is the right place for these operations (unevictable pages are ignored and not accurate ages). What we need is proactive continuous aging and balancing. We need something like, with additions, Multi-gen LRUs or DAMON or page idle tracking for aging and a new mechanism for balancing which takes ages into account.

Multi-gen LRUs will be pretty useful to expose the page warmth in a NUMA node and to target the right page to reclaim for a memcg. We will also need some way to determine how many pages to target in each memcg for a reclaim.

Quoted 8 lines

To give a more concrete example: Let's say we have a system with two memory tiers and multiple low and high priority jobs. For high priority jobs, set the allocation try list from high to low tier and for low priority jobs the reverse of that (I am not sure if we can do that out of the box with today's kernel). In the background we migrate cold memory down the tiers and hot memory in the reverse direction.

Quoted 4 lines

In this background mechanism we can enforce all different limiting policies like Yang's original high and low tier percentage or something like X% of accesses of high priority jobs should be from high tier.

If I understand what you are saying is you desire the kernel to provide the interface to expose performance information like
"X% of accesses of high priority jobs is from high tier", and knobs for user space to tell kernel to re-balance pages on a per job class (or cgroup) basis based on this information. The page re-balancing will be initiated by user space rather than by the kernel, similar to what Wei proposed.

Quoted 4 lines

Basically I am saying until we find from production data that this background mechanism is not strong enough to enforce passive limits, we should delay the decision on limit interfaces.

Implementing hard limit does have a number of rough edges on a per node basis. Probably we should first start with doing the proper accounting and exposing the right performance information.

Tim

#4 lore raw
Quoted 4 lines · Tim Chen

On 6/17/21 11:48 AM, Shakeel Butt wrote:

[...]

Quoted 9 lines
Quoted 8 lines

At the moment "personally" I am more inclined towards a passive approach towards the memcg accounting of memory tiers. By that I mean, let's start by providing a 'usage' interface and get more production/real-world data to motivate the 'limit' interfaces. (One minor reason is that defining the 'limit' interface will force us to make the decision on defining tiers i.e. numa or a set of numa or others).

Quoted 4 lines

Probably we could first start with accounting the memory used in each NUMA node for a cgroup and exposing this information to user space. I think that is useful regardless.

Is memory.numa_stat not good enough? This interface does miss __GFP_ACCOUNT non-slab allocations, percpu and sock.

Quoted 4 lines

There is still a question of whether we want to define a set of numa node or tier and extend the accounting and management at that memory tier abstraction level.

[...]

Quoted 8 lines
Quoted 8 lines

To give a more concrete example: Let's say we have a system with two memory tiers and multiple low and high priority jobs. For high priority jobs, set the allocation try list from high to low tier and for low priority jobs the reverse of that (I am not sure if we can do that out of the box with today's kernel). In the background we migrate cold memory down the tiers and hot memory in the reverse direction.

Quoted 8 lines
Quoted 4 lines

In this background mechanism we can enforce all different limiting policies like Yang's original high and low tier percentage or something like X% of accesses of high priority jobs should be from high tier.

If I understand what you are saying is you desire the kernel to provide the interface to expose performance information like
"X% of accesses of high priority jobs is from high tier",

I think we can estimate "X% of accesses to high tier" using existing perf/PMU counters. So, no new interface.

Quoted 4 lines

and knobs for user space to tell kernel to re-balance pages on a per job class (or cgroup) basis based on this information. The page re-balancing will be initiated by user space rather than by the kernel, similar to what Wei proposed.

This is more open to discussion and we should brainstorm the pros and cons of all proposed approaches.

author
#5 lore raw
Quoted 15 lines · Shakeel Butt
Quoted 4 lines · Tim Chen

On 6/17/21 11:48 AM, Shakeel Butt wrote:

[...]

Quoted 9 lines
Quoted 8 lines

At the moment "personally" I am more inclined towards a passive approach towards the memcg accounting of memory tiers. By that I mean, let's start by providing a 'usage' interface and get more production/real-world data to motivate the 'limit' interfaces. (One minor reason is that defining the 'limit' interface will force us to make the decision on defining tiers i.e. numa or a set of numa or others).

Quoted 6 lines
Quoted 4 lines

Probably we could first start with accounting the memory used in each NUMA node for a cgroup and exposing this information to user space. I think that is useful regardless.

Is memory.numa_stat not good enough?

Yeah, forgot numa_stat is already there. Thanks for reminding me.

Quoted 2 lines

This interface does miss
__GFP_ACCOUNT non-slab allocations, percpu and sock.

numa_stat should be good enough for now.

Quoted 19 lines
Quoted 5 lines

There is still a question of whether we want to define a set of numa node or tier and extend the accounting and management at that memory tier abstraction level.

[...]

Quoted 13 lines
Quoted 12 lines

To give a more concrete example: Let's say we have a system with two memory tiers and multiple low and high priority jobs. For high priority jobs, set the allocation try list from high to low tier and for low priority jobs the reverse of that (I am not sure if we can do that out of the box with today's kernel). In the background we migrate cold memory down the tiers and hot memory in the reverse direction.

In this background mechanism we can enforce all different limiting policies like Yang's original high and low tier percentage or something like X% of accesses of high priority jobs should be from high tier.

Quoted 6 lines
Quoted 3 lines

If I understand what you are saying is you desire the kernel to provide the interface to expose performance information like
"X% of accesses of high priority jobs is from high tier",

I think we can estimate "X% of accesses to high tier" using existing perf/PMU counters. So, no new interface.

Using a perf counter will be okay to do for user space daemon, but I think there will be objections from people that the kernel take away a perf counter to collect perf data in kernel.

Tim

#6 lore raw
Quoted 1 line · Tim Chen

[...]

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

If I understand what you are saying is you desire the kernel to provide the interface to expose performance information like
"X% of accesses of high priority jobs is from high tier",

Quoted 7 lines
Quoted 2 lines

I think we can estimate "X% of accesses to high tier" using existing perf/PMU counters. So, no new interface.

Using a perf counter will be okay to do for user space daemon, but I think there will be objections from people that the kernel take away a perf counter to collect perf data in kernel.

This is one possible policy. I would not focus too much on it unless someone says they want exactly that. In that case we can brainstorm how to provide general infrastructure to enforce such policies. Basically this is like an SLO and the violation triggers the balancing (which can be in use space or kernel and a separate discussion).

#7 lore raw
In reply to Shakeel Butt
Quoted 43 lines · Shakeel Butt

Thanks Yang for the CC.

Quoted 29 lines · Yang Shi
Quoted 27 lines · Tim Chen

From: Tim Chen <[email protected]>

Tiered memory accounting and management
------------------------------------------------------------
Traditionally, all RAM is DRAM. Some DRAM might be closer/faster than others, but a byte of media has about the same cost whether it is close or far. But, with new memory tiers such as High-Bandwidth Memory or Persistent Memory, there is a choice between fast/expensive and slow/cheap. But, the current memory cgroups still live in the old model. There is only one set of limits, and it implies that all memory has the same cost. We would like to extend memory cgroups to comprehend different memory tiers to give users a way to choose a mix between fast/expensive and slow/cheap.

To manage such memory, we will need to account memory usage and impose limits for each kind of memory.

There were a couple of approaches that have been discussed previously to partition the memory between the cgroups listed below. We will like to
use the LSF/MM session to come to a consensus on the approach to take.

1. Per NUMA node limit and accounting for each cgroup.
We can assign higher limits on better performing memory node for higher priority cgroups.

Quoted 9 lines
Quoted 9 lines

There are some loose ends here that warrant further discussions:
(1) A user friendly interface for such limits. Will a proportional weight for the cgroup that translate to actual absolute limit be more suitable?
(2) Memory mis-configurations can occur more easily as the admin has a much larger number of limits spread among between the cgroups to manage. Over-restrictive limits can lead to under utilized and wasted memory and hurt performance.
(3) OOM behavior when a cgroup hits its limit.

Quoted 7 lines

This (numa based limits) is something I was pushing for but after discussing this internally with userspace controller devs, I have to backoff from this position.

The main feedback I got was that setting one memory limit is already complicated and having to set/adjust these many limits would be horrifying.

Yes, that is also what I heard of.

Quoted 42 lines
Quoted 24 lines
Quoted 18 lines

2. Per memory tier limit and accounting for each cgroup.
We can assign higher limits on memories in better performing memory tier for higher priority cgroups. I previously prototyped a soft limit based implementation to demonstrate the tiered limit idea.

There are also a number of issues here:
(1) The advantage is we have fewer limits to deal with simplifying
configuration. However, there are doubts raised by a number of people on whether we can really properly classify the NUMA nodes into memory tiers. There could still be significant performance differences between NUMA nodes even for the same kind of memory. We will also not have the fine-grained control and flexibility that comes with a per NUMA node limit.
(2) Will a memory hierarchy defined by promotion/demotion relationship between
memory nodes be a viable approach for defining memory tiers?

Quoted 2 lines

These issues related to the management of systems with multiple kind of memories can be ironed out in this session.

Thanks for suggesting this topic. I'm interested in the topic and would like to attend.

Quoted 9 lines

Other than the above points. I'm wondering whether we shall discuss "Migrate Pages in lieu of discard" as well? Dave Hansen is driving the development and I have been involved in the early development and review, but it seems there are still some open questions according to the latest review feedback.

Some other folks may be interested in this topic either, CC'ed them in the thread.

At the moment "personally" I am more inclined towards a passive approach towards the memcg accounting of memory tiers. By that I mean, let's start by providing a 'usage' interface and get more production/real-world data to motivate the 'limit' interfaces. (One minor reason is that defining the 'limit' interface will force us to make the decision on defining tiers i.e. numa or a set of numa or others).

Quoted 4 lines

IMHO we should focus more on the "aging" of the application memory and "migration/balance" between the tiers. I don't think the memory reclaim infrastructure is the right place for these operations (unevictable pages are ignored and not accurate ages). What we need is

Why is unevictable pages a problem? I don't get why you have to demote unevictable pages. If you do care what nodes the memory will be mlock'ed on, don't you have to move the memory to the target nodes before mlock them?

Quoted 3 lines

proactive continuous aging and balancing. We need something like, with additions, Multi-gen LRUs or DAMON or page idle tracking for aging and a new mechanism for balancing which takes ages into account.

I agree the better balance could be reached by more accurate aging. It is a more general problem than tier'ed memory specific.

Quoted 6 lines

To give a more concrete example: Let's say we have a system with two memory tiers and multiple low and high priority jobs. For high priority jobs, set the allocation try list from high to low tier and for low priority jobs the reverse of that (I am not sure if we can do that out of the box with today's kernel). In the background we migrate

AFAICT, I don't think we have. With the current APIs, you just can bind to a set of nodes, but the fallback order is one way.

Quoted 8 lines

cold memory down the tiers and hot memory in the reverse direction.

In this background mechanism we can enforce all different limiting policies like Yang's original high and low tier percentage or something like X% of accesses of high priority jobs should be from high tier. Basically I am saying until we find from production data that this background mechanism is not strong enough to enforce passive limits, we should delay the decision on limit interfaces.

#8 lore raw
Quoted 2 lines · Yang Shi

On Thu, Jun 17, 2021 at 11:49 AM Shakeel Butt <[email protected]> wrote:

[...]

Quoted 11 lines
Quoted 5 lines

IMHO we should focus more on the "aging" of the application memory and "migration/balance" between the tiers. I don't think the memory reclaim infrastructure is the right place for these operations (unevictable pages are ignored and not accurate ages). What we need is

Why is unevictable pages a problem? I don't get why you have to demote unevictable pages. If you do care what nodes the memory will be mlock'ed on, don't you have to move the memory to the target nodes before mlock them?

I think we want the ability to balance the memory (hot in higher tier and cold in lower tier) irrespective if it is evictable or not. Similarly we want aging information of both evictable and unevictable memory. If we depend on the reclaim infrastructure for demotion then cold unevictable memory may get stuck in the higher tier and have no aging information of unevictable memory.

Quoted 7 lines
Quoted 3 lines

proactive continuous aging and balancing. We need something like, with additions, Multi-gen LRUs or DAMON or page idle tracking for aging and a new mechanism for balancing which takes ages into account.

I agree the better balance could be reached by more accurate aging. It is a more general problem than tier'ed memory specific.

I agree and proactive reclaim is the other use-case.