psi: fix PSI_MEM_FULL state when tasks are in memstall and doing reclaim
Subsystems (2)
PATCH 1/1 psi: fix PSI_MEM_FULL state when tasks are in memstall and doing reclaim
We've noticed cases where tasks in a cgroup are stalled on memory but there is little memory FULL pressure since tasks stay on the runqueue in reclaim.
A simple example involves a single threaded program that keeps leaking and touching large amounts of memory. It runs in a cgroup with swap enabled, memory.high set at 10M and cpu.max ratio set at 5%. Though there is significant CPU pressure and memory SOME, there is barely any memory FULL since the task enters reclaim and stays on the runqueue. However, this memory-bound task is effectively stalled on memory and we expect memory FULL to match memory SOME in this scenario.
The code is confused about memstall && running, thinking there is a stalled task and a productive task when there's only one task: a reclaimer that's counted as both. To fix this, we redefine the condition for PSI_MEM_FULL to check that all running tasks are in an active memstall instead of checking that there are no running tasks.
case PSI_MEM_FULL:
- return unlikely(tasks[NR_MEMSTALL] && !tasks[NR_RUNNING]);
+ return unlikely(tasks[NR_MEMSTALL] &&
+ tasks[NR_RUNNING] == tasks[NR_MEMSTALL_RUNNING]);
This will capture reclaimers. It will also capture tasks that called psi_memstall_enter() and are about to sleep, but this should be negligible noise.
--- include/linux/psi_types.h | 13 ++++++++++- kernel/sched/psi.c | 45 ++++++++++++++++++++++++--------------- kernel/sched/stats.h | 5 ++++- 3 files changed, 44 insertions(+), 19 deletions(-)
include/linux/psi_types.h +12 -1
@@ -21,7 +21,17 @@ enum psi_task_count {* don't have to special case any state tracking for it. */ NR_ONCPU , - NR_PSI_TASK_COUNTS = 4, +/* +* For IO and CPU stalls the presence of running/oncpu tasks +* in the domain means a partial rather than a full stall. +* For memory it's not so simple because of page reclaimers: +* they are running/oncpu while representing a stall. To tell +* whether a domain has productivity left or not, we need to +* distinguish between regular running (i.e. productive) +* threads and memstall ones. +*/ +NR_MEMSTALL_RUNNING , + NR_PSI_TASK_COUNTS = 5, };/* Task state bitmasks */
@@ -29,6 +39,7 @@ enum psi_task_count {#define TSK_MEMSTALL (1 << NR_MEMSTALL)#define TSK_RUNNING (1 << NR_RUNNING)#define TSK_ONCPU (1 << NR_ONCPU) +#define TSK_MEMSTALL_RUNNING (1 << NR_MEMSTALL_RUNNING)/* Resources that workloads could be stalled on */ enum psi_res {
kernel/sched/psi.c +28 -17
104 lines not on this page · raw message
kernel/sched/stats.h +4 -1
@@ -118,6 +118,9 @@ static inline void psi_enqueue(struct task_struct *p, bool wakeup)if (static_branch_likely (&psi_disabled))return ; +if (p->in_memstall) + set |=TSK_MEMSTALL_RUNNING ; +if (!wakeup || p->sched_psi_wake_requeue) {if (p->in_memstall) set |=TSK_MEMSTALL ;
@@ -148,7 +151,7 @@ static inline void psi_dequeue(struct task_struct *p, bool sleep)return ;if (p->in_memstall) - clear |=TSK_MEMSTALL ; + clear |= (TSK_MEMSTALL |TSK_MEMSTALL_RUNNING );psi_task_change (p, clear,0 ); }
Signature · 1 line
2.30.2
Quoted 23 lines · Brian Chen
We've noticed cases where tasks in a cgroup are stalled on memory but there is little memory FULL pressure since tasks stay on the runqueue in reclaim.
A simple example involves a single threaded program that keeps leaking and touching large amounts of memory. It runs in a cgroup with swap enabled, memory.high set at 10M and cpu.max ratio set at 5%. Though there is significant CPU pressure and memory SOME, there is barely any memory FULL since the task enters reclaim and stays on the runqueue. However, this memory-bound task is effectively stalled on memory and we expect memory FULL to match memory SOME in this scenario.
The code is confused about memstall && running, thinking there is a stalled task and a productive task when there's only one task: a reclaimer that's counted as both. To fix this, we redefine the condition for
PSI_MEM_FULLto check that all running tasks are in an active memstall instead of checking that there are no running tasks.case
PSI_MEM_FULL:kernel/sched/psi.c +2
-1-return unlikely (tasks[NR_MEMSTALL ] && !tasks[NR_RUNNING ]); +return unlikely (tasks[NR_MEMSTALL ] && + tasks[NR_RUNNING ] == tasks[NR_MEMSTALL_RUNNING ]);
Quoted 5 lines
This will capture reclaimers. It will also capture tasks that called
psi_memstall_enter()and are about to sleep, but this should be negligible noise.Signed-off-by Brian Chen <[email protected]>
This bug essentially causes us to count memory-some in walltime and memory-full in tasktime, which can be quite confusing and misleading in combined CPU and memory pressure situations.
The fix looks good to me, thanks Brian.
The bug's been there since the initial psi commit, so I don't think a stable backport is warranted.
Peter, absent objections, can you please pick this up through -tip?
Peter Zijlstra
maintainerQuoted 36 lines · Johannes Weiner
Quoted 23 lines · Brian Chen
We've noticed cases where tasks in a cgroup are stalled on memory but there is little memory FULL pressure since tasks stay on the runqueue in reclaim.
A simple example involves a single threaded program that keeps leaking and touching large amounts of memory. It runs in a cgroup with swap enabled, memory.high set at 10M and cpu.max ratio set at 5%. Though there is significant CPU pressure and memory SOME, there is barely any memory FULL since the task enters reclaim and stays on the runqueue. However, this memory-bound task is effectively stalled on memory and we expect memory FULL to match memory SOME in this scenario.
The code is confused about memstall && running, thinking there is a stalled task and a productive task when there's only one task: a reclaimer that's counted as both. To fix this, we redefine the condition for
PSI_MEM_FULLto check that all running tasks are in an active memstall instead of checking that there are no running tasks.case
PSI_MEM_FULL:kernel/sched/psi.c +2
-1-return unlikely (tasks[NR_MEMSTALL ] && !tasks[NR_RUNNING ]); +return unlikely (tasks[NR_MEMSTALL ] && + tasks[NR_RUNNING ] == tasks[NR_MEMSTALL_RUNNING ]);Quoted 5 lines
This will capture reclaimers. It will also capture tasks that called
psi_memstall_enter()and are about to sleep, but this should be negligible noise.Signed-off-by Brian Chen <[email protected]>Acked-by Johannes Weiner <[email protected]>This bug essentially causes us to count memory-some in walltime and memory-full in tasktime, which can be quite confusing and misleading in combined CPU and memory pressure situations.
Quoted 6 lines
The fix looks good to me, thanks Brian.
The bug's been there since the initial psi commit, so I don't think a stable backport is warranted.
Peter, absent objections, can you please pick this up through -tip?
Yep can do. Note that our psi_group_cpu data structure is now completely filled (the extra tasks state filled the last hole):
struct psi_group_cpu {
seqcount_t seq __attribute__((__aligned__(64))); /* 0 4 */
unsigned int tasks[5]; /* 4 20 */
u32 state_mask; /* 24 4 */
u32 times[7]; /* 28 28 */
u64 state_start; /* 56 8 */
/* --- cacheline 1 boundary (64 bytes) --- */
u32 times_prev[2][7] __attribute__((__aligned__(64))); /* 64 56 */
/* size: 128, cachelines: 2, members: 6 */
/* padding: 8 */
/* forced alignments: 2 */
} __attribute__((__aligned__(64)));[tip: sched/core] psi: Fix PSI_MEM_FULL state when tasks are in memstall and doing reclaim
In reply to Brian ChenThe following commit has been merged into the sched/core branch of tip:
The following commit has been merged into the sched/core branch of tip:
Commit-ID: cb0e52b7748737b2cf6481fdd9b920ce7e1ebbdf Gitweb: https://git.kernel.org/tip/cb0e52b7748737b2cf6481fdd9b920ce7e1ebbdf Author: Brian Chen <[email protected]> AuthorDate: Wed, 10 Nov 2021 21:33:12 Committer: Peter Zijlstra <[email protected]> CommitterDate: Wed, 17 Nov 2021 14:49:00 +01:00
psi: Fix PSI_MEM_FULL state when tasks are in memstall and doing reclaim
We've noticed cases where tasks in a cgroup are stalled on memory but there is little memory FULL pressure since tasks stay on the runqueue in reclaim.
A simple example involves a single threaded program that keeps leaking and touching large amounts of memory. It runs in a cgroup with swap enabled, memory.high set at 10M and cpu.max ratio set at 5%. Though there is significant CPU pressure and memory SOME, there is barely any memory FULL since the task enters reclaim and stays on the runqueue. However, this memory-bound task is effectively stalled on memory and we expect memory FULL to match memory SOME in this scenario.
The code is confused about memstall && running, thinking there is a stalled task and a productive task when there's only one task: a reclaimer that's counted as both. To fix this, we redefine the condition for PSI_MEM_FULL to check that all running tasks are in an active memstall instead of checking that there are no running tasks.
case PSI_MEM_FULL:
- return unlikely(tasks[NR_MEMSTALL] && !tasks[NR_RUNNING]);
+ return unlikely(tasks[NR_MEMSTALL] &&
+ tasks[NR_RUNNING] == tasks[NR_MEMSTALL_RUNNING]);
This will capture reclaimers. It will also capture tasks that called psi_memstall_enter() and are about to sleep, but this should be negligible noise.
--- include/linux/psi_types.h | 13 ++++++++++- kernel/sched/psi.c | 45 +++++++++++++++++++++++--------------- kernel/sched/stats.h | 5 +++- 3 files changed, 44 insertions(+), 19 deletions(-)
include/linux/psi_types.h +12 -1
@@ -22,7 +22,17 @@ enum psi_task_count {* don't have to special case any state tracking for it. */ NR_ONCPU , - NR_PSI_TASK_COUNTS = 4, +/* +* For IO and CPU stalls the presence of running/oncpu tasks +* in the domain means a partial rather than a full stall. +* For memory it's not so simple because of page reclaimers: +* they are running/oncpu while representing a stall. To tell +* whether a domain has productivity left or not, we need to +* distinguish between regular running (i.e. productive) +* threads and memstall ones. +*/ +NR_MEMSTALL_RUNNING , + NR_PSI_TASK_COUNTS = 5, };/* Task state bitmasks */
@@ -30,6 +40,7 @@ enum psi_task_count {#define TSK_MEMSTALL (1 << NR_MEMSTALL)#define TSK_RUNNING (1 << NR_RUNNING)#define TSK_ONCPU (1 << NR_ONCPU) +#define TSK_MEMSTALL_RUNNING (1 << NR_MEMSTALL_RUNNING)/* Resources that workloads could be stalled on */ enum psi_res {
kernel/sched/psi.c +28 -17
104 lines not on this page · raw message
kernel/sched/stats.h +4 -1
@@ -118,6 +118,9 @@ static inline void psi_enqueue(struct task_struct *p, bool wakeup)if (static_branch_likely (&psi_disabled))return ; +if (p->in_memstall) + set |=TSK_MEMSTALL_RUNNING ; +if (!wakeup || p->sched_psi_wake_requeue) {if (p->in_memstall) set |=TSK_MEMSTALL ;
@@ -148,7 +151,7 @@ static inline void psi_dequeue(struct task_struct *p, bool sleep)return ;if (p->in_memstall) - clear |=TSK_MEMSTALL ; + clear |= (TSK_MEMSTALL |TSK_MEMSTALL_RUNNING );psi_task_change (p, clear,0 ); }