New: [bisected] NULL pointer dereference when starting a kvm based VM

Bug 112301 Andrew Morton 4 participants 5 messages Feb 11, 2016 – Feb 17, 2016 1 message not in the archive

Subsystems (3)

Memory management x86 x86 MM

lore

Share & tools

lore

A message that is not in the archive (sent off-list or not yet received): [email protected]/

Andrew Morton

author maintainer

2016-02-11 21:30 UTC #1 lore raw

(switched to email. Please respond via emailed reply-to-all, not via the bugzilla web interface).

Quoted 23 lines · On Thu, 11 Feb 2016 07:09:04 +0000 [email protected] wrote:

https://bugzilla.kernel.org/show_bug.cgi?id=112301
Bug ID: 112301
Summary: [bisected] NULL pointer dereference when starting a
kvm based VM
Product: Memory Management
Version: 2.5
Kernel Version: 4.5-rcX
Hardware: All
OS: Linux
Tree: Mainline
Status: NEW
Severity: normal
Priority: P1
Component: Other
Assignee: [email protected]
Reporter: [email protected]
Regression: No
Created attachment 203451
--> https://bugzilla.kernel.org/attachment.cgi?id=203451&action=edit
Call Trace of a NULL pointer dereference at gup_pte_range

Quoted 7 lines

Starting a qemu-kvm based VM configured to use hughpages I'm getting the following NULL pointer dereference, see attached dmesg section.
The issue was introduced with commit 7d2eba0557c18f7522b98befed98799990dd4fdb
Author: Ebru Akagunduz <[email protected]>
Date:   Thu Jan 14 15:22:19 2016 -0800
mm: add tracepoint for scanning pages

Thanks for the detailed report. Can you please verify that your tree has 629d9d1cafbd49cb374 ("mm: avoid uninitialized variable in tracepoint")?

vfio_pin_pages() doesn't seem to be doing anything crazy. Hugh, Ebru:
could you please take a look?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

[email protected]

2016-02-12 08:26 UTC #2 lore raw

Thanks for the fast reply, I've checked 4.5-rc1 and 4.5-rc3 so the commit 629d9d1cafbd49cb374 was already included. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: [email protected]

Hugh Dickins

2016-02-12 21:10 UTC #3 lore raw

In reply to Andrew Morton

Quoted 37 lines · Andrew Morton

(switched to email. Please respond via emailed reply-to-all, not via the bugzilla web interface).
Quoted 23 lines · On Thu, 11 Feb 2016 07:09:04 +0000 [email protected] wrote:
https://bugzilla.kernel.org/show_bug.cgi?id=112301
Bug ID: 112301
Summary: [bisected] NULL pointer dereference when starting a
kvm based VM
Product: Memory Management
Version: 2.5
Kernel Version: 4.5-rcX
Hardware: All
OS: Linux
Tree: Mainline
Status: NEW
Severity: normal
Priority: P1
Component: Other
Assignee: [email protected]
Reporter: [email protected]
Regression: No
Created attachment 203451
--> https://bugzilla.kernel.org/attachment.cgi?id=203451&action=edit
Call Trace of a NULL pointer dereference at gup_pte_range
Quoted 7 lines
Starting a qemu-kvm based VM configured to use hughpages I'm getting the following NULL pointer dereference, see attached dmesg section.
The issue was introduced with commit 7d2eba0557c18f7522b98befed98799990dd4fdb
Author: Ebru Akagunduz <[email protected]>
Date:   Thu Jan 14 15:22:19 2016 -0800
mm: add tracepoint for scanning pages

Quoted 6 lines

Thanks for the detailed report. Can you please verify that your tree has 629d9d1cafbd49cb374 ("mm: avoid uninitialized variable in tracepoint")?
vfio_pin_pages() doesn't seem to be doing anything crazy. Hugh, Ebru:
could you please take a look?

I very much doubt that the uninitialized variable in collapse_huge_page() had anything to do with the crash in gup_pte_range(). Far more likely is that the bisection hit a point in between the introduction of that uninitialized variable and its subsequent fix, the test crashed, and the bisector didn't notice that it was crashing for a different reason.

Comparing the "Code:" of the gup_pte_range() crash with disassembly of gup_pte_range() here, it looks as if it's crashing in pte_page(). And, yes, that pte_page() looks broken in 4.5-rc: please try this patch.

[PATCH] mm, x86: fix pte_page() crash in gup_pte_range()

Commit 3565fce3a659 ("mm, x86: get_user_pages() for dax mappings") has moved up the pte_page(pte) in x86's fast gup_pte_range(), for no discernible reason: put it back where it belongs, after the pte_flags check and the pfn_valid cross-check.

That may be the cause of the NULL pointer dereference in gup_pte_range(), seen when vfio called vaddr_get_pfn() when starting a qemu-kvm based VM.

Reported-by Michael Long <[email protected]>

Cc Dan Williams <[email protected]>

Signed-off-by Hugh Dickins <[email protected]>

---

 arch/x86/mm/gup.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

arch/x86/mm/gup.c +1 -1

@@ -102,7 +102,6 @@ static noinline int gup_pte_range(pmd_t
 			return 0;
 		}
 
-		page = pte_page(pte);
 		if (pte_devmap(pte)) {
 			pgmap = get_dev_pagemap(pte_pfn(pte), pgmap);
 			if (unlikely(!pgmap)) {

@@ -115,6 +114,7 @@ static noinline int gup_pte_range(pmd_t
 			return 0;
 		}
 		VM_BUG_ON(!pfn_valid(pte_pfn(pte)));
+		page = pte_page(pte);
 		get_page(page);
 		put_dev_pagemap(pgmap);
 		SetPageReferenced(page);

[email protected]

2016-02-13 10:15 UTC #4 lore raw

Quoted 93 lines

Gesendet: Freitag, 12. Februar 2016 um 22:10 Uhr
Von: "Hugh Dickins" <[email protected]>
An: "Andrew Morton" <[email protected]>
Cc: [email protected], [email protected], [email protected], [email protected], "Hugh Dickins" <[email protected]>, "Dan Williams" <[email protected]>, "Ingo Molnar" <[email protected]>, "Thomas Gleixner" <[email protected]>, "H. Peter Anvin" <[email protected]>
Betreff: Re: [Bug 112301] New: [bisected] NULL pointer dereference when starting a kvm based VM
Quoted 37 lines · Andrew Morton
(switched to email. Please respond via emailed reply-to-all, not via the bugzilla web interface).
Quoted 23 lines · On Thu, 11 Feb 2016 07:09:04 +0000 [email protected] wrote:
https://bugzilla.kernel.org/show_bug.cgi?id=112301
Bug ID: 112301
Summary: [bisected] NULL pointer dereference when starting a
kvm based VM
Product: Memory Management
Version: 2.5
Kernel Version: 4.5-rcX
Hardware: All
OS: Linux
Tree: Mainline
Status: NEW
Severity: normal
Priority: P1
Component: Other
Assignee: [email protected]
Reporter: [email protected]
Regression: No
Created attachment 203451
--> https://bugzilla.kernel.org/attachment.cgi?id=203451&action=edit
Call Trace of a NULL pointer dereference at gup_pte_range
Quoted 7 lines
Starting a qemu-kvm based VM configured to use hughpages I'm getting the following NULL pointer dereference, see attached dmesg section.
The issue was introduced with commit 7d2eba0557c18f7522b98befed98799990dd4fdb
Author: Ebru Akagunduz <[email protected]>
Date:   Thu Jan 14 15:22:19 2016 -0800
mm: add tracepoint for scanning pages
Quoted 6 lines
Thanks for the detailed report. Can you please verify that your tree has 629d9d1cafbd49cb374 ("mm: avoid uninitialized variable in tracepoint")?
vfio_pin_pages() doesn't seem to be doing anything crazy. Hugh, Ebru:
could you please take a look?
I very much doubt that the uninitialized variable in collapse_huge_page() had anything to do with the crash in gup_pte_range(). Far more likely is that the bisection hit a point in between the introduction of that uninitialized variable and its subsequent fix, the test crashed, and the bisector didn't notice that it was crashing for a different reason.
Comparing the "Code:" of the gup_pte_range() crash with disassembly of gup_pte_range() here, it looks as if it's crashing in pte_page(). And, yes, that pte_page() looks broken in 4.5-rc: please try this patch.
[PATCH] mm, x86: fix pte_page() crash in gup_pte_range()
Commit 3565fce3a659 ("mm, x86: get_user_pages() for dax mappings") has moved up the pte_page(pte) in x86's fast gup_pte_range(), for no discernible reason: put it back where it belongs, after the pte_flags check and the pfn_valid cross-check.
That may be the cause of the NULL pointer dereference in gup_pte_range(), seen when vfio called vaddr_get_pfn() when starting a qemu-kvm based VM.
Reported-by Michael Long <[email protected]>
Cc Dan Williams <[email protected]>
Signed-off-by Hugh Dickins <[email protected]>
---
 arch/x86/mm/gup.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/mm/gup.c +0 -1
@@ -102,7 +102,6 @@ static noinline int gup_pte_range(pmd_t
 			return 0;
 		}
 
-		page = pte_page(pte);
 		if (pte_devmap(pte)) {
 			pgmap = get_dev_pagemap(pte_pfn(pte), pgmap);
 			if (unlikely(!pgmap)) {

arch/x86/mm/gup.c static noinline int gup_pte_range(pmd_t · line 120

Quoted 7 lines

arch/x86/mm/gup.c +1 -0

@@ -115,6 +114,7 @@ static noinline int gup_pte_range(pmd_t
 			return 0;
 		}
 		VM_BUG_ON(!pfn_valid(pte_pfn(pte)));
+		page = pte_page(pte);
 		get_page(page);
 		put_dev_pagemap(pgmap);
 		SetPageReferenced(page);

Thank gentlemen, the patch solves the issue, everything is back to normal.

Dan Williams

Acked-by

2016-02-17 00:38 UTC #5 lore raw

In reply to Hugh Dickins

Quoted 62 lines · Hugh Dickins

Quoted 37 lines · Andrew Morton
(switched to email. Please respond via emailed reply-to-all, not via the bugzilla web interface).
Quoted 23 lines · On Thu, 11 Feb 2016 07:09:04 +0000 [email protected] wrote:
https://bugzilla.kernel.org/show_bug.cgi?id=112301
Bug ID: 112301
Summary: [bisected] NULL pointer dereference when starting a
kvm based VM
Product: Memory Management
Version: 2.5
Kernel Version: 4.5-rcX
Hardware: All
OS: Linux
Tree: Mainline
Status: NEW
Severity: normal
Priority: P1
Component: Other
Assignee: [email protected]
Reporter: [email protected]
Regression: No
Created attachment 203451
--> https://bugzilla.kernel.org/attachment.cgi?id=203451&action=edit
Call Trace of a NULL pointer dereference at gup_pte_range
Quoted 7 lines
Starting a qemu-kvm based VM configured to use hughpages I'm getting the following NULL pointer dereference, see attached dmesg section.
The issue was introduced with commit 7d2eba0557c18f7522b98befed98799990dd4fdb
Author: Ebru Akagunduz <[email protected]>
Date:   Thu Jan 14 15:22:19 2016 -0800
mm: add tracepoint for scanning pages
Quoted 6 lines
Thanks for the detailed report. Can you please verify that your tree has 629d9d1cafbd49cb374 ("mm: avoid uninitialized variable in tracepoint")?
vfio_pin_pages() doesn't seem to be doing anything crazy. Hugh, Ebru:
could you please take a look?
I very much doubt that the uninitialized variable in collapse_huge_page() had anything to do with the crash in gup_pte_range(). Far more likely is that the bisection hit a point in between the introduction of that uninitialized variable and its subsequent fix, the test crashed, and the bisector didn't notice that it was crashing for a different reason.
Comparing the "Code:" of the gup_pte_range() crash with disassembly of gup_pte_range() here, it looks as if it's crashing in pte_page(). And, yes, that pte_page() looks broken in 4.5-rc: please try this patch.
[PATCH] mm, x86: fix pte_page() crash in gup_pte_range()
Commit 3565fce3a659 ("mm, x86: get_user_pages() for dax mappings") has moved up the pte_page(pte) in x86's fast gup_pte_range(), for no discernible reason: put it back where it belongs, after the pte_flags check and the pfn_valid cross-check.

Quoted 6 lines

That may be the cause of the NULL pointer dereference in gup_pte_range(), seen when vfio called vaddr_get_pfn() when starting a qemu-kvm based VM.
Reported-by Michael Long <[email protected]>
Cc Dan Williams <[email protected]>
Signed-off-by Hugh Dickins <[email protected]>

Acked-by Dan Williams <[email protected]>

That must have been a merge/rebase error on my part when forward porting the patch to a new -mm baseline because the pte_devmap() check is done before we know that the pfn actually has a corresponding struct page.