per-NUMA memory limits in mem cgroup?
Hi,
I'm aware of the ability to use the memory controller to limit how much memory a group of tasks can consume.
Is there any way to limit how much memory a group of tasks can consume *per NUMA node*?
The specific scenario I'm considering is that of a hypervisor host. I have system management stuff running on the host that may need more than one core, and currently these host tasks might be affined to cores from multiple NUMA nodes. I'd like to put a cap on how much memory the host tasks can allocate from each NUMA node in order to ensure that there is a guaranteed amount of memory available for VMs on each NUMA node.
Is this possible, or are the knobs just not there?
Thanks,
Chris
Quoted 7 lines · Chris Friesen
Hi,
I'm aware of the ability to use the memory controller to limit how much memory a group of tasks can consume.
Is there any way to limit how much memory a group of tasks can consume *per NUMA node*?
Not really. We have all or nothing via cpusets but nothing really fine grained for the amount of memory.
Quoted 8 lines
The specific scenario I'm considering is that of a hypervisor host. I have system management stuff running on the host that may need more than one core, and currently these host tasks might be affined to cores from multiple NUMA nodes. I'd like to put a cap on how much memory the host tasks can allocate from each NUMA node in order to ensure that there is a guaranteed amount of memory available for VMs on each NUMA node.
Is this possible, or are the knobs just not there?
Not possible right now. What would be the policy when you reach the limit on one node? Fallback to other nodes? What if those hit the limit as well? OOM killer or an allocation failure?
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 9 lines · Michal Hocko
Quoted 7 lines · Chris Friesen
The specific scenario I'm considering is that of a hypervisor host. I have system management stuff running on the host that may need more than one core, and currently these host tasks might be affined to cores from multiple NUMA nodes. I'd like to put a cap on how much memory the host tasks can allocate from each NUMA node in order to ensure that there is a guaranteed amount of memory available for VMs on each NUMA node.
Quoted 5 lines
Quoted 1 line
Is this possible, or are the knobs just not there?
Not possible right now. What would be the policy when you reach the limit on one node? Fallback to other nodes? What if those hit the limit as well? OOM killer or an allocation failure?
I'd envision it working exactly the same as the current memory cgroup, but with the ability to specify optional per-NUMA-node limits in addition to system-wide.
Chris
Quoted 12 lines · Chris Friesen
Quoted 11 lines · Michal Hocko
Quoted 8 lines · Chris Friesen
The specific scenario I'm considering is that of a hypervisor host. I have system management stuff running on the host that may need more than one core, and currently these host tasks might be affined to cores from multiple NUMA nodes. I'd like to put a cap on how much memory the host tasks can allocate from each NUMA node in order to ensure that there is a guaranteed amount of memory available for VMs on each NUMA node.
Is this possible, or are the knobs just not there?
Quoted 7 lines
Quoted 3 lines
Not possible right now. What would be the policy when you reach the limit on one node? Fallback to other nodes? What if those hit the limit as well? OOM killer or an allocation failure?
I'd envision it working exactly the same as the current memory cgroup, but with the ability to specify optional per-NUMA-node limits in addition to system-wide.
OK, so you would have a per numa percentage of the hard limit? But more importantly, note that the page allocation is done way before the charge so we do not have any control over where the memory get allocated from so we would have to play nasty tricks in the reclaim to somehow balance NUMA charge pools. And I can easily imagine we would go OOM before we saturate all NUMA pools. But I didn't get to think this whole thing through as I am conferencing these days. I am even not sure the whole thing is the best idea as well. It sounds more easily then it would end up, I suspect.
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 17 lines · Michal Hocko
Quoted 16 lines · Chris Friesen
Quoted 14 lines · Michal Hocko
Quoted 8 lines · Chris Friesen
The specific scenario I'm considering is that of a hypervisor host. I have system management stuff running on the host that may need more than one core, and currently these host tasks might be affined to cores from multiple NUMA nodes. I'd like to put a cap on how much memory the host tasks can allocate from each NUMA node in order to ensure that there is a guaranteed amount of memory available for VMs on each NUMA node.
Is this possible, or are the knobs just not there?
Not possible right now. What would be the policy when you reach the limit on one node? Fallback to other nodes? What if those hit the limit as well? OOM killer or an allocation failure?
Quoted 5 lines
Quoted 3 lines
I'd envision it working exactly the same as the current memory cgroup, but with the ability to specify optional per-NUMA-node limits in addition to system-wide.
OK, so you would have a per numa percentage of the hard limit?
I think it'd make more sense as a hard limit per NUMA node.
Quoted 5 lines
But more
importantly, note that the page allocation is done way before the charge so we do not have any control over where the memory get allocated from so we would have to play nasty tricks in the reclaim to somehow balance NUMA charge pools.
Reading the docs on the memory controller it does seem a bit tricky. I had envisioned some sort of "is there memory left in this group" check before "approving" the memory allocation, but it seems it doesn't really work that way.
Chris
On Tue 24-04-18 11:13:16, Chris Friesen wrote:
[...]
Quoted 4 lines
Reading the docs on the memory controller it does seem a bit tricky. I had envisioned some sort of "is there memory left in this group" check before "approving" the memory allocation, but it seems it doesn't really work that way.
No, your memory will be usually eaten by the page cache.
Signature · 2 lines
Michal Hocko SUSE Labs