Skip to contentSkip to conversation

New: BUG: unable to handle kernel NULL pointer dereference under stress (possibly related to https://lkml.org/lkml/2019/5/24/292 ?)

Bug 203715 Andrew Morton 3 participants 10 messages May 29, 2019 – Jun 11, 2019 1 message not in the archive

Subsystems (2)
Memory management Page allocator
lore
Share & tools
lore

Andrew Morton

author maintainer
#1 lore raw
In reply to a message not in the archive

(switched to email. Please respond via emailed reply-to-all, not via the bugzilla web interface).

Mel, we may have a regression from e332f741a8dd1 ("mm, compaction: be selective about what pageblocks to clear skip hints"). The crash sure looks like the one which 60fce36afa9c77c7 ("mm/compaction.c: correct zone boundary handling when isolating pages from a pageblock") fixed, but Gabriele can reproduce it with 5.1.5. I've confirmed that 5.1.5 has 60fce36afa9c77c7.

Thanks.

Quoted 86 lines · On Mon, 27 May 2019 10:12:30 +0000 [email protected] wrote:

https://bugzilla.kernel.org/show_bug.cgi?id=203715

Bug ID: 203715
Summary: BUG: unable to handle kernel NULL pointer dereference
under stress (possibly related to
https://lkml.org/lkml/2019/5/24/292 ?)
Product: Memory Management
Version: 2.5
Kernel Version: 5.1+
Hardware: x86-64
OS: Linux
Tree: Mainline
Status: NEW
Severity: normal
Priority: P1
Component: Page Allocator
Assignee: [email protected]
Reporter: [email protected]
Regression: No

Created attachment 282949
--> https://bugzilla.kernel.org/attachment.cgi?id=282949&action=edit
crash log n.1

hello

since 5.1 I'm getting machine freezes like:

    May  7 18:00:10 dschgrazlin3 kernel: BUG: unable to handle kernel NULL
pointer dereference at 0000000000000000
    May  7 18:00:10 dschgrazlin3 kernel: #PF error: [normal kernel read fault]
    May  7 18:00:10 dschgrazlin3 kernel: PGD 0 P4D 0 
    May  7 18:00:10 dschgrazlin3 kernel: Oops: 0000 [#1] SMP
    May  7 18:00:10 dschgrazlin3 kernel: CPU: 3 PID: 44 Comm: kswapd0 Not
tainted 5.1.0 #1
    May  7 18:00:10 dschgrazlin3 kernel: Hardware name: System manufacturer
System Product Name/F2A85-M PRO, BIOS 5104 09/14/2012
    May  7 18:00:10 dschgrazlin3 kernel: RIP:
0010:__reset_isolation_pfn+0x2cb/0x410
    [...]
    May  7 18:00:10 dschgrazlin3 kernel: Call Trace:
    May  7 18:00:10 dschgrazlin3 kernel:  __reset_isolation_suitable+0x95/0x110
    May  7 18:00:10 dschgrazlin3 kernel:  ? __wake_up_common_lock+0xd0/0xd0
    May  7 18:00:10 dschgrazlin3 kernel:  reset_isolation_suitable+0x34/0x40
    May  7 18:00:10 dschgrazlin3 kernel:  kswapd+0xad/0x2c0
    May  7 18:00:10 dschgrazlin3 kernel:  ? __wake_up_common_lock+0xd0/0xd0
    May  7 18:00:10 dschgrazlin3 kernel:  ? balance_pgdat+0x440/0x440
    May  7 18:00:10 dschgrazlin3 kernel:  kthread+0xff/0x120
    May  7 18:00:10 dschgrazlin3 kernel:  ?
__kthread_create_on_node+0x1b0/0x1b0
    May  7 18:00:10 dschgrazlin3 kernel:  ret_from_fork+0x1f/0x30
    May  7 18:00:10 dschgrazlin3 kernel: CR2: 0000000000000000
    May  7 18:00:10 dschgrazlin3 kernel: ---[ end trace 075fb7a28df7d1d4 ]---
    May  7 18:00:10 dschgrazlin3 kernel: RIP:
0010:__reset_isolation_pfn+0x2cb/0x410
    [...]

(complete logs attached)

I started having this during firefox build, but experienced it during other build processes (mesa, gcc). The problem always appears under heavy load of the machine.

Unfortunately, the problem cannot be triggered with probability=1, although firefox build triggers the machine freeze almost always (at random points of the build, though)

I experience the problem on two twin boxes, which makes me exclude HW issues.

Absolutely no problems when running kernels <5.1 (<=5.0.15)

In some cases, I got the kernel screams without complete machine freeze, but with heavily reduced functionality of the whole system (eg ls command hanging)

Due to the issue not being always reproducible, bisection isn't 100% reliable; however the first bad commit seems to be e332f741a8dd1ec9a6dc8aa997296ecbfe64323e

I'll be happy to provide any other file/information which might be useful

Signature · 2 lines
You are receiving this mail because:
You are the assignee for the bug.
#2 lore raw
Quoted 5 lines · Andrew Morton

(switched to email. Please respond via emailed reply-to-all, not via the bugzilla web interface).

Quoted 7 lines

Mel, we may have a regression from e332f741a8dd1 ("mm, compaction: be selective about what pageblocks to clear skip hints"). The crash sure looks like the one which 60fce36afa9c77c7 ("mm/compaction.c: correct zone boundary handling when isolating pages from a pageblock") fixed, but Gabriele can reproduce it with 5.1.5. I've confirmed that 5.1.5 has 60fce36afa9c77c7.

Sorry, I was on holidays and only playing catchup now. Does this happen to trigger with 5.2-rc3? I ask because there were other fixes in there with stable cc'd that have not been picked up yet. They are a poor match for this particular bug but it would be nice to confirm.

Signature · 2 lines
Mel Gorman
SUSE Labs
#3 lore raw
Quoted 4 lines

Sorry, I was on holidays and only playing catchup now. Does this happen to trigger with 5.2-rc3? I ask because there were other fixes in there with stable cc'd that have not been picked up yet. They are a poor match for this particular bug but it would be nice to confirm.

I'll test 5.2-rc3 as soon as possible and report the results

thanks
ciao
-g

#4 lore raw
In reply to Mel Gorman

hello

Quoted 4 lines

Sorry, I was on holidays and only playing catchup now. Does this happen to trigger with 5.2-rc3? I ask because there were other fixes in there with stable cc'd that have not been picked up yet. They are a poor match for this particular bug but it would be nice to confirm.

I have built v5.2-rc3 from git (stable/linux-stable.git) and tested it against firefox-67.0.1 build: no joy.

I'm going to upload the kernel log and the config I used for v5.2-rc3 (there were a couple of new opts) to bugzilla, if that can help

thanks
ciao
-g

#5 lore raw
Quoted 7 lines · On Wed, Jun 05, 2019 at 02:38:55PM +0200, [email protected] wrote:

hello

Quoted 4 lines

Sorry, I was on holidays and only playing catchup now. Does this happen to trigger with 5.2-rc3? I ask because there were other fixes in there with stable cc'd that have not been picked up yet. They are a poor match for this particular bug but it would be nice to confirm.

Quoted 6 lines

I have built v5.2-rc3 from git (stable/linux-stable.git) and tested it against firefox-67.0.1 build: no joy.

I'm going to upload the kernel log and the config I used for v5.2-rc3 (there were a couple of new opts) to bugzilla, if that can help

Yes, that would be helpful. Thanks.

Signature · 2 lines
Mel Gorman
SUSE Labs
#6 lore raw
In reply to [email protected]
Quoted 7 lines · On Wed, Jun 05, 2019 at 02:38:55PM +0200, [email protected] wrote:

hello

Quoted 4 lines

Sorry, I was on holidays and only playing catchup now. Does this happen to trigger with 5.2-rc3? I ask because there were other fixes in there with stable cc'd that have not been picked up yet. They are a poor match for this particular bug but it would be nice to confirm.

Quoted 6 lines

I have built v5.2-rc3 from git (stable/linux-stable.git) and tested it against firefox-67.0.1 build: no joy.

I'm going to upload the kernel log and the config I used for v5.2-rc3 (there were a couple of new opts) to bugzilla, if that can help

Can you try the following compile-tested only patch please?

mm/compaction.c +1 -2
index 9e1b9acb116b..b3f18084866c 100644
@@ -277,8 +277,7 @@ __reset_isolation_pfn(struct zone *zone, unsigned long pfn, bool check_source,
 	}
 
 	/* Ensure the end of the pageblock or zone is online and valid */
-	block_pfn += pageblock_nr_pages;
-	block_pfn = min(block_pfn, zone_end_pfn(zone) - 1);
+	block_pfn = min(pageblock_end_pfn(block_pfn), zone_end_pfn(zone) - 1);
 	end_page = pfn_to_online_page(block_pfn);
 	if (!end_page)
 		return false;
Signature · 2 lines
Mel Gorman
SUSE Labs
#7 lore raw
Quoted 16 lines

Can you try the following compile-tested only patch please?

mm/compaction.c +1 -2
index 9e1b9acb116b..b3f18084866c 100644
@@ -277,8 +277,7 @@ __reset_isolation_pfn(struct zone *zone, unsigned long pf
 n, bool check_source,
 	}
 
 	/* Ensure the end of the pageblock or zone is online and valid */
-	block_pfn += pageblock_nr_pages;
-	block_pfn = min(block_pfn, zone_end_pfn(zone) - 1);
+	block_pfn = min(pageblock_end_pfn(block_pfn), zone_end_pfn(zone) - 1);
 	end_page = pfn_to_online_page(block_pfn);
 	if (!end_page)
Quoted 2 lines

return false;

Unfortunately it doesn't help: the test firefox build very soon crashed as before; this time the machine froze completely (had to hardware reboot) and I couldn't find any kernel log in the log files (however the screen of the frozen console looked pretty the same as the previous times)

(I applied the patch on top of e577c8b64d58fe307ea4d5149d31615df2d90861, right?)

#8 lore raw
Quoted 10 lines · On Thu, Jun 06, 2019 at 03:20:49PM +0200, [email protected] wrote:
Quoted 10 lines

Can you try the following compile-tested only patch please?

mm/compaction.c +0 -0
index 9e1b9acb116b..b3f18084866c 100644
@@ -277,8 +277,7 @@ __reset_isolation_pfn(struct zone *zone, unsigned long pf
 n, bool check_source,
 	}
Quoted 15 lines
Quoted 8 lines
mm/compaction.c +1 -2
 	/* Ensure the end of the pageblock or zone is online and valid */
-	block_pfn += pageblock_nr_pages;
-	block_pfn = min(block_pfn, zone_end_pfn(zone) - 1);
+	block_pfn = min(pageblock_end_pfn(block_pfn), zone_end_pfn(zone) - 1);
 	end_page = pfn_to_online_page(block_pfn);
 	if (!end_page)

return false;

Unfortunately it doesn't help: the test firefox build very soon crashed as before; this time the machine froze completely (had to hardware reboot) and I couldn't find any kernel log in the log files (however the screen of the frozen console looked pretty the same as the previous times)

Thanks.

Quoted 2 lines

(I applied the patch on top of e577c8b64d58fe307ea4d5149d31615df2d90861, right?)

Please try the following on top of 5.2-rc3

mm/compaction.c +2 -3
index 9e1b9acb116b..69f4ddfddfa4 100644
@@ -277,8 +277,7 @@ __reset_isolation_pfn(struct zone *zone, unsigned long pfn, bool check_source,
 	}
 
 	/* Ensure the end of the pageblock or zone is online and valid */
-	block_pfn += pageblock_nr_pages;
-	block_pfn = min(block_pfn, zone_end_pfn(zone) - 1);
+	block_pfn = min(pageblock_end_pfn(block_pfn), zone_end_pfn(zone) - 1);
 	end_page = pfn_to_online_page(block_pfn);
 	if (!end_page)
 		return false;

@@ -289,7 +288,7 @@ __reset_isolation_pfn(struct zone *zone, unsigned long pfn, bool check_source,
 	 * is necessary for the block to be a migration source/target.
 	 */
 	do {
-		if (pfn_valid_within(pfn)) {
+		if (pfn_valid(pfn)) {
 			if (check_source && PageLRU(page)) {
 				clear_pageblock_skip(page);
 				return true;
#9 lore raw
Quoted 20 lines · Mel Gorman
Quoted 2 lines

(I applied the patch on top of e577c8b64d58fe307ea4d5149d31615df2d90861, right?)

Please try the following on top of 5.2-rc3

mm/compaction.c +1 -2
index 9e1b9acb116b..69f4ddfddfa4 100644
@@ -277,8 +277,7 @@ __reset_isolation_pfn(struct zone *zone, unsigned long pfn, bool check_source,
 	}
 
 	/* Ensure the end of the pageblock or zone is online and valid */
-	block_pfn += pageblock_nr_pages;
-	block_pfn = min(block_pfn, zone_end_pfn(zone) - 1);
+	block_pfn = min(pageblock_end_pfn(block_pfn), zone_end_pfn(zone) - 1);
 	end_page = pfn_to_online_page(block_pfn);
 	if (!end_page)
 		return false;

mm/compaction.c __reset_isolation_pfn(struct zone *zone, unsigned long pfn, bool check_source, · line 294

Quoted 8 lines
mm/compaction.c +1 -1
@@ -289,7 +288,7 @@ __reset_isolation_pfn(struct zone *zone, unsigned long pfn, bool check_source,
 	 * is necessary for the block to be a migration source/target.
 	 */
 	do {
-		if (pfn_valid_within(pfn)) {
+		if (pfn_valid(pfn)) {
 			if (check_source && PageLRU(page)) {
 				clear_pageblock_skip(page);
 				return true;

Any news with this patch?

Signature · 2 lines
Mel Gorman
SUSE Labs
#10 lore raw
Quoted 3 lines · Mel Gorman

Any news with this patch?

oops: I run the patch and reported by email (CC'ing to bugzilla): either I botched something with the reporting mail or you missed the email... (my report is on bugzilla, though, comment 20)

I reproduce the report here:

Quoted 4 lines

no joy; I left the FF build running and found the machine frozen this morning; however, firefox build could apparently complete successfully; I can't say when exactly the problem happened, as I haven't found any message in the logs

I can add that since the last attempt, after rebooting into 5.0.15, I have built a lot of software (including FF) without any problem; this enforces me in the conviction that there must be some problem for kernels >=5.1

Does anybody else reproduce this?

thanks a lot
ciao
-g