Skip to contentSkip to conversation

mm: memcg: Fix memcg reclaim soft lockup

PATCH Superseded Xunlei Pang 4 participants 6 messages Aug 26, 2020

Subsystems (3)

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 mm: memcg: Fix memcg reclaim soft lockup

We've met softlockup with "CONFIG_PREEMPT_NONE=y", when the target memcg doesn't have any reclaimable memory.

It can be easily reproduced as below:

 watchdog: BUG: soft lockup - CPU#0 stuck for 111s![memcg_test:2204]
 CPU: 0 PID: 2204 Comm: memcg_test Not tainted 5.9.0-rc2+ #12
 Call Trace:
  shrink_lruvec+0x49f/0x640
  shrink_node+0x2a6/0x6f0
  do_try_to_free_pages+0xe9/0x3e0
  try_to_free_mem_cgroup_pages+0xef/0x1f0
  try_charge+0x2c1/0x750
  mem_cgroup_charge+0xd7/0x240
  __add_to_page_cache_locked+0x2fd/0x370
  add_to_page_cache_lru+0x4a/0xc0
  pagecache_get_page+0x10b/0x2f0
  filemap_fault+0x661/0xad0
  ext4_filemap_fault+0x2c/0x40
  __do_fault+0x4d/0xf9
  handle_mm_fault+0x1080/0x1790

It only happens on our 1-vcpu instances, because there's no chance for oom reaper to run to reclaim the to-be-killed process.

Add cond_resched() at the upper shrink_node_memcgs() to solve this issue, and any other possible issue like meomry.min protection.

---
 mm/vmscan.c | 2 ++
 1 file changed, 2 insertions(+)
mm/vmscan.c +2 -0
index 99e1796..bbdc38b 100644
@@ -2617,6 +2617,8 @@ static void shrink_node_memcgs(pg_data_t *pgdat, struct scan_control *sc)
 
 		mem_cgroup_calculate_protection(target_memcg, memcg);
 
+		cond_resched();
+
 		if (mem_cgroup_below_min(memcg)) {
 			/*
 			 * Hard protection.
Signature · 1 line
1.8.3.1
#1 lore raw

Hi Xunlei,

Quoted 2 lines · Xunlei Pang

Add cond_resched() at the upper shrink_node_memcgs() to solve this issue, and any other possible issue like meomry.min protection.

I understand the general intent of the patch, but could you clarify your concern around memory protection?

Acked-by
#2 lore raw
Quoted 5 lines · Chris Down
Quoted 2 lines · Xunlei Pang

Add cond_resched() at the upper shrink_node_memcgs() to solve this issue, and any other possible issue like meomry.min protection.

I understand the general intent of the patch, but could you clarify your concern around memory protection?

Oh, I see, your concern is just preemption in general rather than a fixing anything for the memory protection side. In which case, go for it, but I agree with Michael that it would be nice to send v3 with a clarifying comment.

Acked-by on v2 1/1
#3 lore raw
In reply to Xunlei Pang
Quoted 21 lines · Xunlei Pang

We've met softlockup with "CONFIG_PREEMPT_NONE=y", when the target memcg doesn't have any reclaimable memory.

It can be easily reproduced as below:

 watchdog: BUG: soft lockup - CPU#0 stuck for 111s![memcg_test:2204]
 CPU: 0 PID: 2204 Comm: memcg_test Not tainted 5.9.0-rc2+ #12
 Call Trace:
  shrink_lruvec+0x49f/0x640
  shrink_node+0x2a6/0x6f0
  do_try_to_free_pages+0xe9/0x3e0
  try_to_free_mem_cgroup_pages+0xef/0x1f0
  try_charge+0x2c1/0x750
  mem_cgroup_charge+0xd7/0x240
  __add_to_page_cache_locked+0x2fd/0x370
  add_to_page_cache_lru+0x4a/0xc0
  pagecache_get_page+0x10b/0x2f0
  filemap_fault+0x661/0xad0
  ext4_filemap_fault+0x2c/0x40
  __do_fault+0x4d/0xf9
  handle_mm_fault+0x1080/0x1790
Quoted 5 lines

It only happens on our 1-vcpu instances, because there's no chance for oom reaper to run to reclaim the to-be-killed process.

Add cond_resched() at the upper shrink_node_memcgs() to solve this issue, and any other possible issue like meomry.min protection.

I would just add
"
This will mean that we will get a scheduling point for each memcg in the reclaimed hierarchy without any dependency on the reclaimable memory in that memcg thus making it more predictable.
"

Quoted 2 lines

Thanks!

Quoted 19 lines
---
 mm/vmscan.c | 2 ++
 1 file changed, 2 insertions(+)
mm/vmscan.c +2 -0
index 99e1796..bbdc38b 100644
@@ -2617,6 +2617,8 @@ static void shrink_node_memcgs(pg_data_t *pgdat, struct scan_control *sc)
 
 		mem_cgroup_calculate_protection(target_memcg, memcg);
 
+		cond_resched();
+
 		if (mem_cgroup_below_min(memcg)) {
 			/*
 			 * Hard protection.
Signature · 1 line
1.8.3.1
Signature · 2 lines
Michal Hocko
SUSE Labs
maintainer
#4 lore raw
In reply to Xunlei Pang
Quoted 21 lines · Xunlei Pang

We've met softlockup with "CONFIG_PREEMPT_NONE=y", when the target memcg doesn't have any reclaimable memory.

It can be easily reproduced as below:

 watchdog: BUG: soft lockup - CPU#0 stuck for 111s![memcg_test:2204]
 CPU: 0 PID: 2204 Comm: memcg_test Not tainted 5.9.0-rc2+ #12
 Call Trace:
  shrink_lruvec+0x49f/0x640
  shrink_node+0x2a6/0x6f0
  do_try_to_free_pages+0xe9/0x3e0
  try_to_free_mem_cgroup_pages+0xef/0x1f0
  try_charge+0x2c1/0x750
  mem_cgroup_charge+0xd7/0x240
  __add_to_page_cache_locked+0x2fd/0x370
  add_to_page_cache_lru+0x4a/0xc0
  pagecache_get_page+0x10b/0x2f0
  filemap_fault+0x661/0xad0
  ext4_filemap_fault+0x2c/0x40
  __do_fault+0x4d/0xf9
  handle_mm_fault+0x1080/0x1790
Quoted 8 lines

It only happens on our 1-vcpu instances, because there's no chance for oom reaper to run to reclaim the to-be-killed process.

Add cond_resched() at the upper shrink_node_memcgs() to solve this issue, and any other possible issue like meomry.min protection.

This generally makes sense to me but really should have a comment:

/*
* This loop can become CPU-bound when there are thousands
* of cgroups that aren't eligible for reclaim - either
* because they don't have any pages, or because their
* memory is explicitly protected. Avoid soft lockups. */
cond_resched();

The placement in the middle of the multi-part protection checks is a bit odd too. It would be better to have it either at the top of the loop, or at the end, by replacing the continues with goto next.

#5 lore raw
Quoted 41 lines · Johannes Weiner
Quoted 21 lines · Xunlei Pang

We've met softlockup with "CONFIG_PREEMPT_NONE=y", when the target memcg doesn't have any reclaimable memory.

It can be easily reproduced as below:

 watchdog: BUG: soft lockup - CPU#0 stuck for 111s![memcg_test:2204]
 CPU: 0 PID: 2204 Comm: memcg_test Not tainted 5.9.0-rc2+ #12
 Call Trace:
  shrink_lruvec+0x49f/0x640
  shrink_node+0x2a6/0x6f0
  do_try_to_free_pages+0xe9/0x3e0
  try_to_free_mem_cgroup_pages+0xef/0x1f0
  try_charge+0x2c1/0x750
  mem_cgroup_charge+0xd7/0x240
  __add_to_page_cache_locked+0x2fd/0x370
  add_to_page_cache_lru+0x4a/0xc0
  pagecache_get_page+0x10b/0x2f0
  filemap_fault+0x661/0xad0
  ext4_filemap_fault+0x2c/0x40
  __do_fault+0x4d/0xf9
  handle_mm_fault+0x1080/0x1790
Quoted 8 lines

It only happens on our 1-vcpu instances, because there's no chance for oom reaper to run to reclaim the to-be-killed process.

Add cond_resched() at the upper shrink_node_memcgs() to solve this issue, and any other possible issue like meomry.min protection.

This generally makes sense to me but really should have a comment:

/*
* This loop can become CPU-bound when there are thousands
* of cgroups that aren't eligible for reclaim - either
* because they don't have any pages, or because their
* memory is explicitly protected. Avoid soft lockups. */
cond_resched();

Quoted 3 lines

The placement in the middle of the multi-part protection checks is a bit odd too. It would be better to have it either at the top of the loop, or at the end, by replacing the continues with goto next.

Yes makes sense. I would stick it to the begining of the loop to make it stand out and make it obvious wrt code flow.

Signature · 2 lines
Michal Hocko
SUSE Labs