Skip to contentSkip to conversation

mm/page_alloc: fix watchdog soft lockups during set_zone_contiguous()

PATCH Applied David Hildenbrand 3 participants 4 messages Apr 16, 2020 – Apr 20, 2020

Subsystems (2)
Memory management Page allocator
lore
Share & tools
lore

David Hildenbrand

author
#0 lore raw

PATCH v2 1/1 mm/page_alloc: fix watchdog soft lockups during set_zone_contiguous()

Without CONFIG_PREEMPT, it can happen that we get soft lockups detected, e.g., while booting up.

[  105.608900] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [swapper/0:1]
[  105.608933] Modules linked in:
[  105.608933] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.6.0-next-20200331+ #4
[  105.608933] Hardware name: Red Hat KVM, BIOS 1.11.1-4.module+el8.1.0+4066+0f1aadab 04/01/2014
[  105.608933] RIP: 0010:__pageblock_pfn_to_page+0x134/0x1c0
[  105.608933] Code: 85 c0 74 71 4a 8b 04 d0 48 85 c0 74 68 48 01 c1 74 63 f6 01 04 74 5e 48 c1 e7 06 4c 8b 05 cc 991
[  105.608933] RSP: 0000:ffffb6d94000fe60 EFLAGS: 00010286 ORIG_RAX: ffffffffffffff13
[  105.608933] RAX: fffff81953250000 RBX: 000000000a4c9600 RCX: ffff8fe9ff7c1990
[  105.608933] RDX: ffff8fe9ff7dab80 RSI: 000000000a4c95ff RDI: 0000000293250000
[  105.608933] RBP: ffff8fe9ff7dab80 R08: fffff816c0000000 R09: 0000000000000008
[  105.608933] R10: 0000000000000014 R11: 0000000000000014 R12: 0000000000000000
[  105.608933] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[  105.608933] FS:  0000000000000000(0000) GS:ffff8fe1ff400000(0000) knlGS:0000000000000000
[  105.608933] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  105.608933] CR2: 000000000f613000 CR3: 00000088cf20a000 CR4: 00000000000006f0
[  105.608933] Call Trace:
[  105.608933]  set_zone_contiguous+0x56/0x70
[  105.608933]  page_alloc_init_late+0x166/0x176
[  105.608933]  kernel_init_freeable+0xfa/0x255
[  105.608933]  ? rest_init+0xaa/0xaa
[  105.608933]  kernel_init+0xa/0x106
[  105.608933]  ret_from_fork+0x35/0x40

The issue becomes visible when having a lot of memory (e.g., 4TB) assigned to a single NUMA node - a system that can easily be created using QEMU. Inside VMs on a hypervisor with quite some memory overcommit, this is fairly easy to trigger.

Reviewed-by Pavel Tatashin <[email protected]>
Reviewed-by Pankaj Gupta <[email protected]>
Reviewed-by Baoquan He <[email protected]>
Reviewed-by Shile Zhang <[email protected]>
Acked-by Michal Hocko <[email protected]>
Cc Andrew Morton <[email protected]>
Cc Kirill Tkhai <[email protected]>
Cc Shile Zhang <[email protected]>
Cc Pavel Tatashin <[email protected]>
Cc Daniel Jordan <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Alexander Duyck <[email protected]>
Cc Baoquan He <[email protected]>
Cc Oscar Salvador <[email protected]>
Signed-off-by David Hildenbrand <[email protected]>

---

This is just a resend of patch #2 of
https://lkml.kernel.org/r/[email protected]

Not that patch #1 was replaced by
https://lkml.kernel.org/r/[email protected]

---
 mm/page_alloc.c | 1 +
 1 file changed, 1 insertion(+)
mm/page_alloc.c +1 -0
index 1385d786a01a..ae1545246b69 100644
@@ -1607,6 +1607,7 @@ void set_zone_contiguous(struct zone *zone)
 		if (!__pageblock_pfn_to_page(block_start_pfn,
 					     block_end_pfn, zone))
 			return;
+		cond_resched();
 	}
 
 	/* We confirm that there is no hole */
Signature · 1 line
2.25.1
maintainer
#1 lore raw
Quoted 8 lines · On Thu, 16 Apr 2020 09:34:17 +0200 David Hildenbrand <[email protected]> wrote:

Without CONFIG_PREEMPT, it can happen that we get soft lockups detected, e.g., while booting up.

...

mm/page_alloc.c void set_zone_contiguous(struct zone *zone) · line 1613

Quoted 7 lines
mm/page_alloc.c +1 -0
@@ -1607,6 +1607,7 @@ void set_zone_contiguous(struct zone *zone)
 		if (!__pageblock_pfn_to_page(block_start_pfn,
 					     block_end_pfn, zone))
 			return;
+		cond_resched();
 	}
 
 	/* We confirm that there is no hole */

I added cc:stable to this one. Please let me know if that wasn't a good idea.

#2 lore raw
Quoted 16 lines · Andrew Morton
Quoted 14 lines · On Thu, 16 Apr 2020 09:34:17 +0200 David Hildenbrand <[email protected]> wrote:

Without CONFIG_PREEMPT, it can happen that we get soft lockups detected, e.g., while booting up.

...

mm/page_alloc.c +1 -0
@@ -1607,6 +1607,7 @@ void set_zone_contiguous(struct zone *zone)
 		if (!__pageblock_pfn_to_page(block_start_pfn,
 					     block_end_pfn, zone))
 			return;
+		cond_resched();
 	}
Quoted 4 lines

mm/page_alloc.c void set_zone_contiguous(struct zone *zone) · line 1613

Quoted 1 line
mm/page_alloc.c +0 -0
 	/* We confirm that there is no hole */

I added cc:stable to this one. Please let me know if that wasn't a good idea.

Really large memory setups tend to run on distribution kernels so backporting to old kernels doesn't really harm.

Signature · 2 lines
Michal Hocko
SUSE Labs
author
#3 lore raw
Quoted 17 lines · Michal Hocko
Quoted 16 lines · Andrew Morton
Quoted 14 lines · On Thu, 16 Apr 2020 09:34:17 +0200 David Hildenbrand <[email protected]> wrote:

Without CONFIG_PREEMPT, it can happen that we get soft lockups detected, e.g., while booting up.

...

mm/page_alloc.c +1 -0
@@ -1607,6 +1607,7 @@ void set_zone_contiguous(struct zone *zone)
 		if (!__pageblock_pfn_to_page(block_start_pfn,
 					     block_end_pfn, zone))
 			return;
+		cond_resched();
 	}
Quoted 8 lines
Quoted 4 lines

mm/page_alloc.c void set_zone_contiguous(struct zone *zone) · line 1613

Quoted 1 line
mm/page_alloc.c +0 -0
 	/* We confirm that there is no hole */

I added cc:stable to this one. Please let me know if that wasn't a good idea.

Really large memory setups tend to run on distribution kernels so backporting to old kernels doesn't really harm.

Yeah, shouldn't hurt.

Signature · 3 lines
Thanks,

David / dhildenb