mm: memcg: add priority for soft limit reclaiming
Subsystems (3)
Quoted 16 lines · On Mon, 23 Sep 2019 21:28:34 Michal Hocko wrote:
Quoted 14 lines · Hillf Danton
Quoted 12 lines · On Thu, 19 Sep 2019 21:32:31 +0800 Michal Hocko wrote:
Quoted 9 lines · Hillf Danton
Currently memory controler is playing increasingly important role in how memory is used and how pages are reclaimed on memory pressure.
In daily works memcg is often created for critical tasks and their pre configured memory usage is supposed to be met even on memory pressure. Administrator wants to make it configurable that the pages consumed by memcg-B can be reclaimed by page allocations invoked not by memcg-A but by memcg-C.
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
I am not really sure I understand the usecase well but this sounds like what memory reclaim protection in v2 is aiming at.
Please describe the usecase.
It is for quite a while that task-A has been able to preempt task-B for cpu cycles. IOW the physical resource cpu cycles are preemptible.
Are physical pages are preemptible too in the same manner? Nope without priority defined for pages currently (say the link between page->nice and task->nice).
The slrp is added for memcg instead of nice because 1) it is only used in the page reclaiming context (in memcg it is soft limit reclaiming), and 2) it is difficult to compare reclaimer and reclaimee task->nice directly in that context as only info about reclaimer and lru page is available.
Here task->nice is replaced with memcg->slrp in order to do page preemption, PP. There is no way for task-A to PP task-B, but the group containing task-A can PP the group containing task-B. That preemption needs code within 100 lines as you see on top of the current memory controller framework.
The user visible things following PP include
1) the increase in system-wide configurability,
Combined with and/or in parallel to memcg.high, PP help admin configure and maintain 100 mm groups on systems with 100GB RAM. With every group high bundary set to 10MB, then he only needs to fiddle with the slrps of handful of groups containing critical tasks.
2) the increase in system-wide responsibility,
Because critical groups can be configured to be not page preempted.
3) the gradient field grows in a long running system with prioirty,
Just like the rivers going through all the ways from mountains to the seas.
Adding PP in background reclaiming is on the way:
1> define page->nice and link it to task->nice
2> on isolating lru pages check reclaimer->nice against page->nice
and skip page if reclaimer is lower on priority
Quoted 4 lines
Quoted 1 line
A tipoint to the v2 stuff please.
Documentation/admin-guide/cgroup-v2.rst
Thanks Michal.
Out of surprise slrp happened to go with the line of cgroup-v2.
Documentation/admin-guide/cgroup-v2.rst +11 -0
@@ -1108,6 +1108,17 @@ PAGE_SIZE multiple when read back. Going over the high limit never invokes the OOM killer and under extreme conditions the limit may be breached. + memory.slrp + A read-write single value [0-32] file which exists on non-root + cgroups. The default is "0". + + Soft limit reclaiming priority. This is the mechanism to control + how physical pages are reclaimed when a group's memory usage goes + over its high boundary. + + It makes sure that no pages will be reclaimed from any group of + higher slrp in favor of a lower-slrp group. + memory.max A read-write single value file which exists on non-root cgroups. The default is "max".
--
Hillf
Quoted 36 lines · Hillf Danton
Quoted 16 lines · On Mon, 23 Sep 2019 21:28:34 Michal Hocko wrote:
Quoted 14 lines · Hillf Danton
Quoted 12 lines · On Thu, 19 Sep 2019 21:32:31 +0800 Michal Hocko wrote:
Quoted 9 lines · Hillf Danton
Currently memory controler is playing increasingly important role in how memory is used and how pages are reclaimed on memory pressure.
In daily works memcg is often created for critical tasks and their pre configured memory usage is supposed to be met even on memory pressure. Administrator wants to make it configurable that the pages consumed by memcg-B can be reclaimed by page allocations invoked not by memcg-A but by memcg-C.
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
I am not really sure I understand the usecase well but this sounds like what memory reclaim protection in v2 is aiming at.
Please describe the usecase.
It is for quite a while that task-A has been able to preempt task-B for cpu cycles. IOW the physical resource cpu cycles are preemptible.
Are physical pages are preemptible too in the same manner? Nope without priority defined for pages currently (say the link between
page->niceandtask->nice).The slrp is added for memcg instead of nice because 1) it is only used in the page reclaiming context (in memcg it is soft limit reclaiming), and 2) it is difficult to compare reclaimer and reclaimee
task->nicedirectly in that context as only info about reclaimer and lru page is available.
Quoted 5 lines
Here
task->niceis replaced withmemcg->slrpin order to do page preemption, PP. There is no way for task-A to PP task-B, but the group containing task-A can PP the group containing task-B. That preemption needs code within 100 lines as you see on top of the current memory controller framework.
This is exactly what the reclaim protection in memcg v2 is meant to be used for. Also soft limit reclaim is absolutely terrible to achieve that because it is just too gross to result in any smooth experience (just have a look how it is doing priority 0 scannig!).
I am not going to even go further wrt the implementation because I belive the priority is even semantically broken wrt hierarchical behavior.
But really, make sure you look into the existing feature set that memcg v2 provides already and come back if you find it unsuitable and we can move from there. Soft limit reclaim is dead and we should let it RIP.
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 48 lines · Michal Hocko
Quoted 36 lines · Hillf Danton
Quoted 16 lines · On Mon, 23 Sep 2019 21:28:34 Michal Hocko wrote:
Quoted 14 lines · Hillf Danton
Quoted 12 lines · On Thu, 19 Sep 2019 21:32:31 +0800 Michal Hocko wrote:
Quoted 9 lines · Hillf Danton
Currently memory controler is playing increasingly important role in how memory is used and how pages are reclaimed on memory pressure.
In daily works memcg is often created for critical tasks and their pre configured memory usage is supposed to be met even on memory pressure. Administrator wants to make it configurable that the pages consumed by memcg-B can be reclaimed by page allocations invoked not by memcg-A but by memcg-C.
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines
I am not really sure I understand the usecase well but this sounds like what memory reclaim protection in v2 is aiming at.
Please describe the usecase.
It is for quite a while that task-A has been able to preempt task-B for cpu cycles. IOW the physical resource cpu cycles are preemptible.
Are physical pages are preemptible too in the same manner? Nope without priority defined for pages currently (say the link between
page->niceandtask->nice).The slrp is added for memcg instead of nice because 1) it is only used in the page reclaiming context (in memcg it is soft limit reclaiming), and 2) it is difficult to compare reclaimer and reclaimee
task->nicedirectly in that context as only info about reclaimer and lru page is available.Quoted 5 lines
Here
task->niceis replaced withmemcg->slrpin order to do page preemption, PP. There is no way for task-A to PP task-B, but the group containing task-A can PP the group containing task-B. That preemption needs code within 100 lines as you see on top of the current memory controller framework.This is exactly what the reclaim protection in memcg v2 is meant to be used for. Also soft limit reclaim is absolutely terrible to achieve that because it is just too gross to result in any smooth experience (just have a look how it is doing priority 0 scannig!).
Quoted 7 lines
I am not going to even go further wrt the implementation because I belive the priority is even semantically broken wrt hierarchical behavior.
But really, make sure you look into the existing feature set that memcg v2 provides already and come back if you find it unsuitable and we can move from there. Soft limit reclaim is dead and we should let it RIP.
Can't agree more here.
Cgroup v2 memory protection mechanisms (memory.low/min) should perfectly solve the described problem. If not, let's fix them rather than extend soft reclaim which is already dead.
Thanks!
On Tue, 24 Sep 2019 17:23:35 +0000 from Roman Gushchin
Quoted 7 lines · Michal Hocko
Quoted 4 lines
But really, make sure you look into the existing feature set that memcg v2 provides already and come back if you find it unsuitable and we can move from there. Soft limit reclaim is dead and we should let it RIP.
Quoted 6 lines
Can't agree more here.
Cgroup v2 memory protection mechanisms (memory.low/min) should perfectly solve the described problem. If not, let's fix them rather than extend soft reclaim which is already dead.
Hehe, IIUC memory.low/min is essentially drawing a line that reclaimers would try their best not to cross. Page preemption OTOH is near ten miles away from that line though it is now on the shoulder of soft reclaiming.
Quoted 11 lines · Hillf Danton
On Tue, 24 Sep 2019 17:23:35 +0000 from Roman Gushchin
Quoted 9 lines · Michal Hocko
Quoted 4 lines
But really, make sure you look into the existing feature set that memcg v2 provides already and come back if you find it unsuitable and we can move from there. Soft limit reclaim is dead and we should let it RIP.
Can't agree more here.
Quoted 7 lines
Quoted 4 lines
Cgroup v2 memory protection mechanisms (memory.low/min) should perfectly solve the described problem. If not, let's fix them rather than extend soft reclaim which is already dead.
Hehe, IIUC memory.low/min is essentially drawing a line that reclaimers would try their best not to cross. Page preemption OTOH is near ten miles away from that line though it is now on the shoulder of soft reclaiming.
Dynamic low limit tuning would achieve exactly what you are after - aka prioritizing some memory consumers over others.
Signature · 2 lines
Michal Hocko SUSE Labs