Skip to contentSkip to conversation

New: handle_mm_fault / xen_pmd_val / radix_tree_lookup_slot Null pointer

Bug 198497 Andrew Morton 6 participants 16 messages Jan 18, 2018 – Apr 12, 2018 1 message not in the archive

Subsystems (4)
lore
Share & tools
lore

Andrew Morton

author maintainer
#1 lore raw
In reply to a message not in the archive

(switched to email. Please respond via emailed reply-to-all, not via the bugzilla web interface).

Quoted 12 lines · On Thu, 18 Jan 2018 01:21:56 +0000 [email protected] wrote:

https://bugzilla.kernel.org/show_bug.cgi?id=198497

Bug ID: 198497
Summary: handle_mm_fault / xen_pmd_val / radix_tree_lookup_slot
Null pointer
Product: Memory Management
Version: 2.5
Kernel Version: Linux app1.vpsgate.com
4.14.13-rh10-20180115190010.xenU.i386 #1 SMP Mon Jan 15 19:04:55 UTC 2018 i686 GNU/Linux
Hardware: Intel
OS: Linux

Quoted 8 lines

Tree: Mainline
Status: NEW
Severity: high
Priority: P1
Component: Page Allocator
Assignee: [email protected]
Reporter: [email protected]
Regression: No

Does this look familiar to anyone?

Quoted 90 lines

On a Xen VM running as pvh

[    3.499843] Adding 131068k swap on /dev/xvda9.  Priority:-2 extents:1
across:131068k SSFS
[    3.547312] EXT4-fs (xvda1): re-mounted. Opts: (null)
[    3.988606] EXT4-fs (xvda1): re-mounted. Opts: errors=remount-ro
[   24.647744] BUG: unable to handle kernel NULL pointer dereference at
00000008
[   24.647801] IP: __radix_tree_lookup+0x14/0xa0
[   24.647811] *pdpt = 00000000253d6027 *pde = 0000000000000000 
[   24.647828] Oops: 0000 [#1] SMP
[   24.647842] CPU: 5 PID: 3600 Comm: java Not tainted
4.14.13-rh10-20180115190010.xenU.i386 #1
[   24.647855] task: e52518c0 task.stack: e4e7a000
[   24.647866] EIP: __radix_tree_lookup+0x14/0xa0
[   24.647876] EFLAGS: 00010286 CPU: 5
[   24.647884] EAX: 00000004 EBX: 00000007 ECX: 00000000 EDX: 00000000
[   24.647895] ESI: 00000000 EDI: 00000000 EBP: e4e7bdb8 ESP: e4e7bda0
[   24.647904]  DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0069
[   24.647917] CR0: 80050033 CR2: 00000008 CR3: 25360000 CR4: 00002660
[   24.647930] Call Trace:
[   24.647942]  radix_tree_lookup_slot+0x13/0x30
[   24.647955]  find_get_entry+0x1d/0x120
[   24.647963]  pagecache_get_page+0x1f/0x230
[   24.647975]  lookup_swap_cache+0x42/0x140
[   24.647983]  swap_readahead_detect+0x66/0x2e0
[   24.647993]  do_swap_page+0x1fa/0x860
[   24.648010]  ? __raw_callee_save___pv_queued_spin_unlock+0x9/0x10
[   24.648026]  ? xen_pmd_val+0x10/0x20
[   24.648035]  handle_mm_fault+0x6f8/0x1020
[   24.648046]  __do_page_fault+0x18a/0x450
[   24.648055]  ? vmalloc_sync_all+0x250/0x250
[   24.648063]  do_page_fault+0x21/0x30
[   24.648074]  common_exception+0x45/0x4a
[   24.648082] EIP: 0xb76d873e
[   24.648088] EFLAGS: 00010206 CPU: 5
[   24.648096] EAX: 76a10000 EBX: 76a1cd14 ECX: 00000006 EDX: 00000006
[   24.648105] ESI: 00000040 EDI: b796c380 EBP: 77881008 ESP: 77880ff8
[   24.648115]  DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 007b
[   24.648124] Code: ff ff ff 00 47 03 e9 69 ff ff ff 8b 45 08 89 06 e9 1f ff
ff ff 66 90 55 89 e5 57 89 d7 56 53 83 ec 0c 89 45 ec 89 4d e8 8b 45 ec <8b> 58
04 89 d8 83 e0 03 48 89 5d f0 75 64 89 d8 83 e0 fe 0f b6
[   24.648195] EIP: __radix_tree_lookup+0x14/0xa0 SS:ESP: 0069:e4e7bda0
[   24.648205] CR2: 0000000000000008
[   24.648273] ---[ end trace ed356e59f215ce07 ]---
[   28.890326] BUG: unable to handle kernel NULL pointer dereference at
00000008
[   28.890372] IP: __radix_tree_lookup+0x14/0xa0
[   28.890382] *pdpt = 0000000025488027 *pde = 0000000000000000 
[   28.890396] Oops: 0000 [#2] SMP
[   28.890408] CPU: 7 PID: 3542 Comm: java Tainted: G      D        
4.14.13-rh10-20180115190010.xenU.i386 #1
[   28.890423] task: e8691080 task.stack: e52a6000
[   28.890433] EIP: __radix_tree_lookup+0x14/0xa0
[   28.890442] EFLAGS: 00010286 CPU: 7
[   28.890449] EAX: 00000004 EBX: 00000007 ECX: 00000000 EDX: 00000000
[   28.890459] ESI: 00000000 EDI: 00000000 EBP: e52a7db8 ESP: e52a7da0
[   28.890469]  DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0069
[   28.890484] CR0: 80050033 CR2: 00000008 CR3: 25161000 CR4: 00002660
[   28.890498] Call Trace:
[   28.890510]  radix_tree_lookup_slot+0x13/0x30
[   28.890522]  find_get_entry+0x1d/0x120
[   28.890531]  pagecache_get_page+0x1f/0x230
[   28.890541]  lookup_swap_cache+0x42/0x140
[   28.890550]  swap_readahead_detect+0x66/0x2e0
[   28.890559]  do_swap_page+0x1fa/0x860
[   28.890573]  ? __raw_callee_save___pv_queued_spin_unlock+0x9/0x10
[   28.890588]  ? xen_pmd_val+0x10/0x20
[   28.890597]  handle_mm_fault+0x6f8/0x1020
[   28.890607]  __do_page_fault+0x18a/0x450
[   28.890616]  ? vmalloc_sync_all+0x250/0x250
[   28.890681]  do_page_fault+0x21/0x30
[   28.890707]  common_exception+0x45/0x4a
[   28.890715] EIP: 0xb779774f
[   28.890722] EFLAGS: 00010202 CPU: 7
[   28.890730] EAX: 00000000 EBX: 66dd9d6c ECX: 02000000 EDX: 00000001
[   28.890740] ESI: 02000000 EDI: 00000000 EBP: 674fe068 ESP: 674fe058
[   28.890751]  DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 007b
[   28.890759] Code: ff ff ff 00 47 03 e9 69 ff ff ff 8b 45 08 89 06 e9 1f ff
ff ff 66 90 55 89 e5 57 89 d7 56 53 83 ec 0c 89 45 ec 89 4d e8 8b 45 ec <8b> 58
04 89 d8 83 e0 03 48 89 5d f0 75 64 89 d8 83 e0 fe 0f b6
[   28.890830] EIP: __radix_tree_lookup+0x14/0xa0 SS:ESP: 0069:e52a7da0
[   28.890841] CR2: 0000000000000008
[   28.890886] ---[ end trace ed356e59f215ce08 ]---
# java -version
java version "1.7.0_51"
Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
Java HotSpot(TM) Server VM (build 24.51-b03, mixed mode)
Quoted 12 lines

~# free -m

             total       used       free     shared    buffers     cached
Mem:          5418        572       4846          0         25        245
-/+ buffers/cache:        301       5117

Swap: 127 0 127

# uptime
 01:21:08 up  2:02,  3 users,  load average: 13.47, 13.65, 13.63
Signature · 2 lines
You are receiving this mail because:
You are the assignee for the bug.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#2 lore raw
Quoted 28 lines · Andrew Morton

(switched to email. Please respond via emailed reply-to-all, not via the bugzilla web interface).

Quoted 12 lines · On Thu, 18 Jan 2018 01:21:56 +0000 [email protected] wrote:

https://bugzilla.kernel.org/show_bug.cgi?id=198497

Bug ID: 198497
Summary: handle_mm_fault / xen_pmd_val / radix_tree_lookup_slot
Null pointer
Product: Memory Management
Version: 2.5
Kernel Version: Linux app1.vpsgate.com
4.14.13-rh10-20180115190010.xenU.i386 #1 SMP Mon Jan 15 19:04:55 UTC 2018 i686 GNU/Linux
Hardware: Intel
OS: Linux

Quoted 8 lines

Tree: Mainline
Status: NEW
Severity: high
Priority: P1
Component: Page Allocator
Assignee: [email protected]
Reporter: [email protected]
Regression: No

Quoted 2 lines

Does this look familiar to anyone?

Fedora has been seeing similar reports
https://bugzilla.redhat.com/show_bug.cgi?id=1531779

Multiple reporters, one in XEN, another on actual hardware

Quoted 103 lines
Quoted 90 lines

On a Xen VM running as pvh

[    3.499843] Adding 131068k swap on /dev/xvda9.  Priority:-2 extents:1
across:131068k SSFS
[    3.547312] EXT4-fs (xvda1): re-mounted. Opts: (null)
[    3.988606] EXT4-fs (xvda1): re-mounted. Opts: errors=remount-ro
[   24.647744] BUG: unable to handle kernel NULL pointer dereference at
00000008
[   24.647801] IP: __radix_tree_lookup+0x14/0xa0
[   24.647811] *pdpt = 00000000253d6027 *pde = 0000000000000000
[   24.647828] Oops: 0000 [#1] SMP
[   24.647842] CPU: 5 PID: 3600 Comm: java Not tainted
4.14.13-rh10-20180115190010.xenU.i386 #1
[   24.647855] task: e52518c0 task.stack: e4e7a000
[   24.647866] EIP: __radix_tree_lookup+0x14/0xa0
[   24.647876] EFLAGS: 00010286 CPU: 5
[   24.647884] EAX: 00000004 EBX: 00000007 ECX: 00000000 EDX: 00000000
[   24.647895] ESI: 00000000 EDI: 00000000 EBP: e4e7bdb8 ESP: e4e7bda0
[   24.647904]  DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0069
[   24.647917] CR0: 80050033 CR2: 00000008 CR3: 25360000 CR4: 00002660
[   24.647930] Call Trace:
[   24.647942]  radix_tree_lookup_slot+0x13/0x30
[   24.647955]  find_get_entry+0x1d/0x120
[   24.647963]  pagecache_get_page+0x1f/0x230
[   24.647975]  lookup_swap_cache+0x42/0x140
[   24.647983]  swap_readahead_detect+0x66/0x2e0
[   24.647993]  do_swap_page+0x1fa/0x860
[   24.648010]  ? __raw_callee_save___pv_queued_spin_unlock+0x9/0x10
[   24.648026]  ? xen_pmd_val+0x10/0x20
[   24.648035]  handle_mm_fault+0x6f8/0x1020
[   24.648046]  __do_page_fault+0x18a/0x450
[   24.648055]  ? vmalloc_sync_all+0x250/0x250
[   24.648063]  do_page_fault+0x21/0x30
[   24.648074]  common_exception+0x45/0x4a
[   24.648082] EIP: 0xb76d873e
[   24.648088] EFLAGS: 00010206 CPU: 5
[   24.648096] EAX: 76a10000 EBX: 76a1cd14 ECX: 00000006 EDX: 00000006
[   24.648105] ESI: 00000040 EDI: b796c380 EBP: 77881008 ESP: 77880ff8
[   24.648115]  DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 007b
[   24.648124] Code: ff ff ff 00 47 03 e9 69 ff ff ff 8b 45 08 89 06 e9 1f ff
ff ff 66 90 55 89 e5 57 89 d7 56 53 83 ec 0c 89 45 ec 89 4d e8 8b 45 ec <8b> 58
04 89 d8 83 e0 03 48 89 5d f0 75 64 89 d8 83 e0 fe 0f b6
[   24.648195] EIP: __radix_tree_lookup+0x14/0xa0 SS:ESP: 0069:e4e7bda0
[   24.648205] CR2: 0000000000000008
[   24.648273] ---[ end trace ed356e59f215ce07 ]---
[   28.890326] BUG: unable to handle kernel NULL pointer dereference at
00000008
[   28.890372] IP: __radix_tree_lookup+0x14/0xa0
[   28.890382] *pdpt = 0000000025488027 *pde = 0000000000000000
[   28.890396] Oops: 0000 [#2] SMP
[   28.890408] CPU: 7 PID: 3542 Comm: java Tainted: G      D
4.14.13-rh10-20180115190010.xenU.i386 #1
[   28.890423] task: e8691080 task.stack: e52a6000
[   28.890433] EIP: __radix_tree_lookup+0x14/0xa0
[   28.890442] EFLAGS: 00010286 CPU: 7
[   28.890449] EAX: 00000004 EBX: 00000007 ECX: 00000000 EDX: 00000000
[   28.890459] ESI: 00000000 EDI: 00000000 EBP: e52a7db8 ESP: e52a7da0
[   28.890469]  DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0069
[   28.890484] CR0: 80050033 CR2: 00000008 CR3: 25161000 CR4: 00002660
[   28.890498] Call Trace:
[   28.890510]  radix_tree_lookup_slot+0x13/0x30
[   28.890522]  find_get_entry+0x1d/0x120
[   28.890531]  pagecache_get_page+0x1f/0x230
[   28.890541]  lookup_swap_cache+0x42/0x140
[   28.890550]  swap_readahead_detect+0x66/0x2e0
[   28.890559]  do_swap_page+0x1fa/0x860
[   28.890573]  ? __raw_callee_save___pv_queued_spin_unlock+0x9/0x10
[   28.890588]  ? xen_pmd_val+0x10/0x20
[   28.890597]  handle_mm_fault+0x6f8/0x1020
[   28.890607]  __do_page_fault+0x18a/0x450
[   28.890616]  ? vmalloc_sync_all+0x250/0x250
[   28.890681]  do_page_fault+0x21/0x30
[   28.890707]  common_exception+0x45/0x4a
[   28.890715] EIP: 0xb779774f
[   28.890722] EFLAGS: 00010202 CPU: 7
[   28.890730] EAX: 00000000 EBX: 66dd9d6c ECX: 02000000 EDX: 00000001
[   28.890740] ESI: 02000000 EDI: 00000000 EBP: 674fe068 ESP: 674fe058
[   28.890751]  DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 007b
[   28.890759] Code: ff ff ff 00 47 03 e9 69 ff ff ff 8b 45 08 89 06 e9 1f ff
ff ff 66 90 55 89 e5 57 89 d7 56 53 83 ec 0c 89 45 ec 89 4d e8 8b 45 ec <8b> 58
04 89 d8 83 e0 03 48 89 5d f0 75 64 89 d8 83 e0 fe 0f b6
[   28.890830] EIP: __radix_tree_lookup+0x14/0xa0 SS:ESP: 0069:e52a7da0
[   28.890841] CR2: 0000000000000008
[   28.890886] ---[ end trace ed356e59f215ce08 ]---
# java -version
java version "1.7.0_51"
Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
Java HotSpot(TM) Server VM (build 24.51-b03, mixed mode)
Quoted 12 lines

~# free -m

              total       used       free     shared    buffers     cached
Mem:          5418        572       4846          0         25        245
-/+ buffers/cache:        301       5117

Swap: 127 0 127

# uptime
  01:21:08 up  2:02,  3 users,  load average: 13.47, 13.65, 13.63
Signature · 2 lines
You are receiving this mail because:
You are the assignee for the bug.
Quoted 6 lines

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#3 lore raw
Quoted 32 lines · Laura Abbott
Quoted 31 lines · Andrew Morton
Quoted 31 lines
[   24.647744] BUG: unable to handle kernel NULL pointer dereference at
00000008
[   24.647801] IP: __radix_tree_lookup+0x14/0xa0
[   24.647811] *pdpt = 00000000253d6027 *pde = 0000000000000000
[   24.647828] Oops: 0000 [#1] SMP
[   24.647842] CPU: 5 PID: 3600 Comm: java Not tainted
4.14.13-rh10-20180115190010.xenU.i386 #1
[   24.647855] task: e52518c0 task.stack: e4e7a000
[   24.647866] EIP: __radix_tree_lookup+0x14/0xa0
[   24.647876] EFLAGS: 00010286 CPU: 5
[   24.647884] EAX: 00000004 EBX: 00000007 ECX: 00000000 EDX: 00000000
[   24.647895] ESI: 00000000 EDI: 00000000 EBP: e4e7bdb8 ESP: e4e7bda0
[   24.647904]  DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0069
[   24.647917] CR0: 80050033 CR2: 00000008 CR3: 25360000 CR4: 00002660
[   24.647930] Call Trace:
[   24.647942]  radix_tree_lookup_slot+0x13/0x30
[   24.647955]  find_get_entry+0x1d/0x120
[   24.647963]  pagecache_get_page+0x1f/0x230
[   24.647975]  lookup_swap_cache+0x42/0x140
[   24.647983]  swap_readahead_detect+0x66/0x2e0
[   24.647993]  do_swap_page+0x1fa/0x860
[   24.648010]  ? __raw_callee_save___pv_queued_spin_unlock+0x9/0x10
[   24.648026]  ? xen_pmd_val+0x10/0x20
[   24.648035]  handle_mm_fault+0x6f8/0x1020
[   24.648046]  __do_page_fault+0x18a/0x450
[   24.648055]  ? vmalloc_sync_all+0x250/0x250
[   24.648063]  do_page_fault+0x21/0x30
[   24.648074]  common_exception+0x45/0x4a
[   24.648082] EIP: 0xb76d873e
[   24.648088] EFLAGS: 00010206 CPU: 5
[   24.648096] EAX: 76a10000 EBX: 76a1cd14 ECX: 00000006 EDX: 00000006
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
[   24.648105] ESI: 00000040 EDI: b796c380 EBP: 77881008 ESP: 77880ff8
[   24.648115]  DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 007b
[   24.648124] Code: ff ff ff 00 47 03 e9 69 ff ff ff 8b 45 08 89 06 e9 1f ff
ff ff 66 90 55 89 e5 57 89 d7 56 53 83 ec 0c 89 45 ec 89 4d e8 8b 45 ec <8b> 58
04 89 d8 83 e0 03 48 89 5d f0 75 64 89 d8 83 e0 fe 0f b6
[   24.648195] EIP: __radix_tree_lookup+0x14/0xa0 SS:ESP: 0069:e4e7bda0
[   24.648205] CR2: 0000000000000008
[   24.648273] ---[ end trace ed356e59f215ce07 ]---

Running that code through decodecode, I get:

   0:	55                   	push   %ebp
   1:	89 e5                	mov    %esp,%ebp
   3:	57                   	push   %edi
   4:	89 d7                	mov    %edx,%edi
   6:	56                   	push   %esi
   7:	53                   	push   %ebx
   8:	83 ec 0c             	sub    $0xc,%esp
   b:	89 45 ec             	mov    %eax,-0x14(%ebp)
   e:	89 4d e8             	mov    %ecx,-0x18(%ebp)
  11:	8b 45 ec             	mov    -0x14(%ebp),%eax
  14:*	8b 58 04             	mov    0x4(%eax),%ebx		<-- trapping instruction
  17:	89 d8                	mov    %ebx,%eax
  19:	83 e0 03             	and    $0x3,%eax

Which I think means it's looking at offset 4 from whichever argument the x86 calling convention puts in register %eax. Which I think is argument 0? Which is the radix tree root. And that makes sense; we're loading the root node from the radix tree root at offset 4. The problem is that %eax has the value 4 in it. That would match with 'page_tree' being at offset 4 from the start of address_space. So find_get_page() got called with a NULL mapping, so pagecache_get_page() got called with a NULL mapping.

Which means I've tracked it back to:

        page = find_get_page(swap_address_space(entry), swp_offset(entry));

and swap_address_space() is returning NULL. Has this machine run swapoff recently, perhaps?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#4 lore raw
Quoted 66 lines · Matthew Wilcox
Quoted 32 lines · Laura Abbott
Quoted 31 lines · Andrew Morton
Quoted 31 lines
[   24.647744] BUG: unable to handle kernel NULL pointer dereference at
00000008
[   24.647801] IP: __radix_tree_lookup+0x14/0xa0
[   24.647811] *pdpt = 00000000253d6027 *pde = 0000000000000000
[   24.647828] Oops: 0000 [#1] SMP
[   24.647842] CPU: 5 PID: 3600 Comm: java Not tainted
4.14.13-rh10-20180115190010.xenU.i386 #1
[   24.647855] task: e52518c0 task.stack: e4e7a000
[   24.647866] EIP: __radix_tree_lookup+0x14/0xa0
[   24.647876] EFLAGS: 00010286 CPU: 5
[   24.647884] EAX: 00000004 EBX: 00000007 ECX: 00000000 EDX: 00000000
[   24.647895] ESI: 00000000 EDI: 00000000 EBP: e4e7bdb8 ESP: e4e7bda0
[   24.647904]  DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0069
[   24.647917] CR0: 80050033 CR2: 00000008 CR3: 25360000 CR4: 00002660
[   24.647930] Call Trace:
[   24.647942]  radix_tree_lookup_slot+0x13/0x30
[   24.647955]  find_get_entry+0x1d/0x120
[   24.647963]  pagecache_get_page+0x1f/0x230
[   24.647975]  lookup_swap_cache+0x42/0x140
[   24.647983]  swap_readahead_detect+0x66/0x2e0
[   24.647993]  do_swap_page+0x1fa/0x860
[   24.648010]  ? __raw_callee_save___pv_queued_spin_unlock+0x9/0x10
[   24.648026]  ? xen_pmd_val+0x10/0x20
[   24.648035]  handle_mm_fault+0x6f8/0x1020
[   24.648046]  __do_page_fault+0x18a/0x450
[   24.648055]  ? vmalloc_sync_all+0x250/0x250
[   24.648063]  do_page_fault+0x21/0x30
[   24.648074]  common_exception+0x45/0x4a
[   24.648082] EIP: 0xb76d873e
[   24.648088] EFLAGS: 00010206 CPU: 5
[   24.648096] EAX: 76a10000 EBX: 76a1cd14 ECX: 00000006 EDX: 00000006
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
[   24.648105] ESI: 00000040 EDI: b796c380 EBP: 77881008 ESP: 77880ff8
[   24.648115]  DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 007b
[   24.648124] Code: ff ff ff 00 47 03 e9 69 ff ff ff 8b 45 08 89 06 e9 1f ff
ff ff 66 90 55 89 e5 57 89 d7 56 53 83 ec 0c 89 45 ec 89 4d e8 8b 45 ec <8b> 58
04 89 d8 83 e0 03 48 89 5d f0 75 64 89 d8 83 e0 fe 0f b6
[   24.648195] EIP: __radix_tree_lookup+0x14/0xa0 SS:ESP: 0069:e4e7bda0
[   24.648205] CR2: 0000000000000008
[   24.648273] ---[ end trace ed356e59f215ce07 ]---

Running that code through decodecode, I get:

    0:	55                   	push   %ebp
    1:	89 e5                	mov    %esp,%ebp
    3:	57                   	push   %edi
    4:	89 d7                	mov    %edx,%edi
    6:	56                   	push   %esi
    7:	53                   	push   %ebx
    8:	83 ec 0c             	sub    $0xc,%esp
    b:	89 45 ec             	mov    %eax,-0x14(%ebp)
    e:	89 4d e8             	mov    %ecx,-0x18(%ebp)
   11:	8b 45 ec             	mov    -0x14(%ebp),%eax
   14:*	8b 58 04             	mov    0x4(%eax),%ebx		<-- trapping instruction
   17:	89 d8                	mov    %ebx,%eax
   19:	83 e0 03             	and    $0x3,%eax

Which I think means it's looking at offset 4 from whichever argument the x86 calling convention puts in register %eax. Which I think is argument 0? Which is the radix tree root. And that makes sense; we're loading the root node from the radix tree root at offset 4. The problem is that %eax has the value 4 in it. That would match with 'page_tree' being at offset 4 from the start of address_space. So find_get_page() got called with a NULL mapping, so pagecache_get_page() got called with a NULL mapping.

Quoted 6 lines

Which means I've tracked it back to:

         page = find_get_page(swap_address_space(entry), swp_offset(entry));

and swap_address_space() is returning NULL. Has this machine run swapoff recently, perhaps?

Swap was on.A Swap is small (127MB).A Swap had not been dipped into.
A A A A A A A A A A A A totalA A A A A A usedA A A A A A freeA A A A sharedA A A buffers cached Swap:A A A A A A A A A 127A A A A A A A A A 0A A A A A A A 127

PS: cannot recall seeing this issue on x86_64, just 32 bit.A PPS: reminder this is on a Xen VM which per https://xenbits.xen.org/docs/unstable/man/xl.cfg.5.html#PVH-Guest-Specific-Options has "out of sync pagetables" if that is relevant (we do not set that option, I am unsure what default is used).

#5 lore raw
Quoted 4 lines · On Fri, Jan 19, 2018 at 04:14:42PM +1300, [email protected] wrote:
Quoted 2 lines · Matthew Wilcox
Quoted 1 line · Laura Abbott

On 01/18/2018 01:55 PM, Andrew Morton wrote:

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
[   24.647744] BUG: unable to handle kernel NULL pointer dereference at
00000008
[   24.647801] IP: __radix_tree_lookup+0x14/0xa0
[   24.647811] *pdpt = 00000000253d6027 *pde = 0000000000000000
[   24.647828] Oops: 0000 [#1] SMP
[   24.647842] CPU: 5 PID: 3600 Comm: java Not tainted
4.14.13-rh10-20180115190010.xenU.i386 #1
[   24.647855] task: e52518c0 task.stack: e4e7a000
[   24.647866] EIP: __radix_tree_lookup+0x14/0xa0
[   24.647876] EFLAGS: 00010286 CPU: 5
[   24.647884] EAX: 00000004 EBX: 00000007 ECX: 00000000 EDX: 00000000

If my understanding is right, EDX contains the index we're looking up. Which is zero. So the swp_entry we got is one bit away from being NULL. Hmm. Have you run memtest86 or some other memory tester on the system recently?

Quoted 1 line

PS: cannot recall seeing this issue on x86_64, just 32 bit.

Laura has 64-bit instances of this.

PPS: reminder

Quoted 3 lines

this is on a Xen VM which per https://xenbits.xen.org/docs/unstable/man/xl.cfg.5.html#PVH-Guest-Specific-Options
has "out of sync pagetables" if that is relevant (we do not set that option,
I am unsure what default is used).

Laura also has non-Xen instances of this. They may not all be the same bug, of course.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#16 lore raw
In reply to Laura Abbott
Quoted 4 lines · Laura Abbott

Fedora has been seeing similar reports
https://bugzilla.redhat.com/show_bug.cgi?id=1531779

Multiple reporters, one in XEN, another on actual hardware

Can you chuck this patch into Fedora? Should make it easier to see if it's a "stuck bit" kind of a problem.

---

From: Matthew Wilcox <[email protected]>
Subject: Detect bad swap entries in lookup

If we have a stuck bit in a PTE, we can end up looking for an entry in a NULL mapping, which oopses fairly quickly. Print a warning to help us debug, and return NULL which will help the machine survive a little longer. Although if it has a permanently stuck bit in a PTE, there's only a 50% chance it'll surive the insertion of a real PTE into that entry.

mm/swap_state.c +5 -1
index 39ae7cfad90f..5a928e0191a1 100644
@@ -334,8 +334,12 @@ struct page *lookup_swap_cache(swp_entry_t entry, struct vm_area_struct *vma,
 	struct page *page;
 	unsigned long ra_info;
 	int win, hits, readahead;
+	struct address_space *swapper_space = swap_address_space(entry);
+
+	if (WARN(!swapper_space, "Bad swp_entry: %lx\n", entry.val))
+		return NULL;
 
-	page = find_get_page(swap_address_space(entry), swp_offset(entry));
+	page = find_get_page(swapper_space, swp_offset(entry));
 
 	INC_CACHE_INFO(find_total);
 	if (page) {

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#6 lore raw
In reply to Matthew Wilcox
Quoted 17 lines · Matthew Wilcox
Quoted 4 lines · On Fri, Jan 19, 2018 at 04:14:42PM +1300, [email protected] wrote:
Quoted 2 lines · Matthew Wilcox
Quoted 1 line · Laura Abbott

On 01/18/2018 01:55 PM, Andrew Morton wrote:

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
[   24.647744] BUG: unable to handle kernel NULL pointer dereference at
00000008
[   24.647801] IP: __radix_tree_lookup+0x14/0xa0
[   24.647811] *pdpt = 00000000253d6027 *pde = 0000000000000000
[   24.647828] Oops: 0000 [#1] SMP
[   24.647842] CPU: 5 PID: 3600 Comm: java Not tainted
4.14.13-rh10-20180115190010.xenU.i386 #1
[   24.647855] task: e52518c0 task.stack: e4e7a000
[   24.647866] EIP: __radix_tree_lookup+0x14/0xa0
[   24.647876] EFLAGS: 00010286 CPU: 5
[   24.647884] EAX: 00000004 EBX: 00000007 ECX: 00000000 EDX: 00000000
Quoted 9 lines

If my understanding is right, EDX contains the index we're looking up. Which is zero. So the swp_entry we got is one bit away from being NULL. Hmm. Have you run memtest86 or some other memory tester on the system recently?

Quoted 1 line

PS: cannot recall seeing this issue on x86_64, just 32 bit.

Laura has 64-bit instances of this.

The 64-bit backtraces reported in the bugzilla looked different, I would consider it a different issue.

Quoted 8 lines

PPS: reminder

Quoted 3 lines

this is on a Xen VM which per https://xenbits.xen.org/docs/unstable/man/xl.cfg.5.html#PVH-Guest-Specific-Options
has "out of sync pagetables" if that is relevant (we do not set that option,
I am unsure what default is used).

Laura also has non-Xen instances of this. They may not all be the same bug, of course.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#7 lore raw
Quoted 23 lines · Laura Abbott
Quoted 22 lines · Matthew Wilcox
Quoted 4 lines · On Fri, Jan 19, 2018 at 04:14:42PM +1300, [email protected] wrote:
Quoted 2 lines · Matthew Wilcox
Quoted 1 line · Laura Abbott

On 01/18/2018 01:55 PM, Andrew Morton wrote:

Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines
Quoted 11 lines

[A A 24.647744] BUG: unable to handle kernel NULL pointer dereference at 00000008

[A A  24.647801] IP: __radix_tree_lookup+0x14/0xa0
[A A  24.647811] *pdpt = 00000000253d6027 *pde = 0000000000000000
[A A  24.647828] Oops: 0000 [#1] SMP
[A A  24.647842] CPU: 5 PID: 3600 Comm: java Not tainted
4.14.13-rh10-20180115190010.xenU.i386 #1
[A A  24.647855] task: e52518c0 task.stack: e4e7a000
[A A  24.647866] EIP: __radix_tree_lookup+0x14/0xa0
[A A  24.647876] EFLAGS: 00010286 CPU: 5
[A A  24.647884] EAX: 00000004 EBX: 00000007 ECX: 00000000 EDX: 00000000

If my understanding is right, EDX contains the index we're looking up. Which is zero.A So the swp_entry we got is one bit away from being NULL. Hmm.A Have you run memtest86 or some other memory tester on the system recently?

Quoted 16 lines
Quoted 4 lines
Quoted 1 line

PS: cannot recall seeing this issue on x86_64, just 32 bit.

Laura has 64-bit instances of this.

The 64-bit backtraces reported in the bugzilla looked different, I would consider it a different issue.

Quoted 8 lines

PPS: reminder

Quoted 3 lines

this is on a Xen VM which per https://xenbits.xen.org/docs/unstable/man/xl.cfg.5.html#PVH-Guest-Specific-Options
has "out of sync pagetables" if that is relevant (we do not set that option,
I am unsure what default is used).

Laura also has non-Xen instances of this.A They may not all be the same bug, of course.

Re-tried with the current latest 4.14 (4.14.15).A Received the following:

[2018-01-24 19:26:57] Ubuntu 14.04.5 LTS dev hvc0
[2018-01-24 19:26:57]
[2018-01-24 19:26:57] dev login: [44501.106868] BUG: unable to handle kernel NULL pointer dereference at 00000008
[2018-01-25 07:47:50] [44501.106897] IP: __radix_tree_lookup+0x14/0xa0
[2018-01-25 07:47:50] [44501.106905] *pdpt = 000000001fe82027 *pde = 0000000000000000
[2018-01-25 07:47:50] [44501.106916] Oops: 0000 [#1] SMP
[2018-01-25 07:47:50] [44501.106924] CPU: 0 PID: 3344 Comm: PassengerAgent Not tainted 4.14.15-rh13-20180123235331.xenU.i386 #1
[2018-01-25 07:47:50] [44501.106935] task: dfee39c0 task.stack: dff12000
[2018-01-25 07:47:50] [44501.106943] EIP: __radix_tree_lookup+0x14/0xa0
[2018-01-25 07:47:50] [44501.106950] EFLAGS: 00210286 CPU: 0
[2018-01-25 07:47:50] [44501.106955] EAX: 00000004 EBX: 00000001 ECX: 00000000 EDX: 00000000
[2018-01-25 07:47:50] [44501.106963] ESI: 00000000 EDI: 00000000 EBP: dff13db8 ESP: dff13da0
[2018-01-25 07:47:50] [44501.106971] A DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0069
[2018-01-25 07:47:50] [44501.106979] CR0: 80050033 CR2: 00000008 CR3: 1fdb1000 CR4: 00002660
[2018-01-25 07:47:50] [44501.106989] Call Trace:
[2018-01-25 07:47:50] [44501.106995] A radix_tree_lookup_slot+0x13/0x30
[2018-01-25 07:47:50] [44501.107004] A find_get_entry+0x1d/0x120
[2018-01-25 07:47:50] [44501.107011] A pagecache_get_page+0x1f/0x230
[2018-01-25 07:47:50] [44501.107018] A lookup_swap_cache+0x42/0x140
[2018-01-25 07:47:50] [44501.107024] A swap_readahead_detect+0x66/0x2e0
[2018-01-25 07:47:50] [44501.107032] A do_swap_page+0x1fa/0x860
[2018-01-25 07:47:50] [44501.107040] A ? __raw_callee_save___pv_queued_spin_unlock+0x9/0x10
[2018-01-25 07:47:50] [44501.107050] A ? xen_pmd_val+0x10/0x20
[2018-01-25 07:47:50] [44501.107057] A handle_mm_fault+0x6f8/0x1020
[2018-01-25 07:47:50] [44501.107065] A ? _raw_spin_unlock_irqrestore+0x13/0x20
[2018-01-25 07:47:50] [44501.107074] A ? pvclock_clocksource_read+0xa6/0x1a0
[2018-01-25 07:47:50] [44501.107081] A __do_page_fault+0x18a/0x450
[2018-01-25 07:47:50] [44501.107089] A ? _copy_to_user+0x28/0x40
[2018-01-25 07:47:50] [44501.107096] A ? vmalloc_sync_all+0x250/0x250
[2018-01-25 07:47:50] [44501.107102] A do_page_fault+0x21/0x30
[2018-01-25 07:47:50] [44501.107109] A common_exception+0x45/0x4a
[2018-01-25 07:47:50] [44501.107115] EIP: 0x82c3358
[2018-01-25 07:47:50] [44501.107120] EFLAGS: 00210202 CPU: 0
[2018-01-25 07:47:50] [44501.107126] EAX: b702d0b8 EBX: 081557a9 ECX: 00000000 EDX: 0a4296bc
[2018-01-25 07:47:50] [44501.107133] ESI: b467c2cc EDI: 00000000 EBP: b467c138 ESP: b467c110
[2018-01-25 07:47:50] [44501.107141] A DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 007b
[2018-01-25 07:47:50] [44501.107147] Code: ff ff ff 00 47 03 e9 69 ff ff ff 8b 45 08 89 06 e9 1f ff ff ff 66 90 55 89 e5 57 89 d7 56 53 83 ec 0c 89 45 ec 89 4d e8 8b 45 ec <8b> 58 04 89 d8 83 e0 03 48 89 5d f0 75 64 89 d8 83 e0 fe 0f b6 [2018-01-25 07:47:50] [44501.110296] EIP: __radix_tree_lookup+0x14/0xa0 SS:ESP: 0069:dff13da0
[2018-01-25 07:47:50] [44501.110304] CR2: 0000000000000008
[2018-01-25 07:47:50] [44501.110356] ---[ end trace 89cdd2ba8e7323a8 ]---

#8 lore raw
Quoted 5 lines · On Fri, Jan 26, 2018 at 07:54:06PM +1300, [email protected] wrote:

Re-tried with the current latest 4.14 (4.14.15). Received the following:

[2018-01-24 19:26:57] dev login: [44501.106868] BUG: unable to handle kernel NULL pointer dereference at 00000008
[2018-01-25 07:47:50] [44501.106897] IP: __radix_tree_lookup+0x14/0xa0

Please try including this patch:

https://bugzilla.kernel.org/show_bug.cgi?id=198497#c7

And have you had the chance to run memtest86 yet?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#9 lore raw
Quoted 11 lines · Matthew Wilcox
Quoted 5 lines · On Fri, Jan 26, 2018 at 07:54:06PM +1300, [email protected] wrote:

Re-tried with the current latest 4.14 (4.14.15).A Received the following:

[2018-01-24 19:26:57] dev login: [44501.106868] BUG: unable to handle kernel NULL pointer dereference at 00000008
[2018-01-25 07:47:50] [44501.106897] IP: __radix_tree_lookup+0x14/0xa0

Please try including this patch:

https://bugzilla.kernel.org/show_bug.cgi?id=198497#c7

And have you had the chance to run memtest86 yet?

Added the patch at https://bugzilla.kernel.org/show_bug.cgi?id=198497#c7

After, received this stack.

Have not tried memtest86.A These are production hosts.A This has occurred on multiple hosts.A I can only recall this occurring on 32 bit kernels.A I cannot recall issues with other VMs not running that kernel on the same hosts.

[ A 125.329163] Bad swp_entry: e000000
[ A 125.329202] ------------[ cut here ]------------
[ A 125.329219] WARNING: CPU: 0 PID: 4175 at mm/swap_state.c:339 lookup_swap_cache+0x140/0x160
[ A 125.329233] CPU: 0 PID: 4175 Comm: apt-show-versio Not tainted 4.14.15-rh14-20180126233810.xenU.i386-00001-g6ba70cb #1
[ A 125.329245] task: ead9a940 task.stack: e7c8c000
[ A 125.329253] EIP: lookup_swap_cache+0x140/0x160
[ A 125.329260] EFLAGS: 00010282 CPU: 0
[ A 125.329267] EAX: 00000016 EBX: 00000000 ECX: ec5289c4 EDX: 0100016d
[ A 125.329275] ESI: b6312000 EDI: e7d94ea0 EBP: e7c8de24 ESP: e7c8de0c
[ A 125.329284] A DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0069
[ A 125.329295] CR0: 80050033 CR2: b63124b0 CR3: 2718c000 CR4: 00002660
[ A 125.329308] Call Trace:
[ A 125.329323] A ? percpu_counter_add_batch+0x91/0xb0
[ A 125.329332] A swap_readahead_detect+0x66/0x2e0
[ A 125.329343] A ? radix_tree_tag_set+0x7a/0xe0
[ A 125.329352] A do_swap_page+0x1fa/0x860
[ A 125.329361] A ? __set_page_dirty_buffers+0xb1/0xe0
[ A 125.329372] A ? ext4_set_page_dirty+0x22/0x60
[ A 125.329383] A ? fault_dirty_shared_page.isra.90+0x3e/0xa0
[ A 125.329396] A ? xen_pmd_val+0x10/0x20
[ A 125.329403] A handle_mm_fault+0x6f8/0x1020
[ A 125.329414] A ? handle_irq_event_percpu+0x3c/0x50
[ A 125.329424] A __do_page_fault+0x18a/0x450
[ A 125.329432] A ? vmalloc_sync_all+0x250/0x250
[ A 125.329439] A do_page_fault+0x21/0x30
[ A 125.329449] A common_exception+0x45/0x4a
[ A 125.329456] EIP: 0xb7ce397b
[ A 125.329462] EFLAGS: 00010202 CPU: 0
[ A 125.329469] EAX: 0000052a EBX: b7d77ff4 ECX: 000004fa EDX: b6311000
[ A 125.329477] ESI: bf90eae0 EDI: b6ed4b20 EBP: bf90ea60 ESP: bf90ea20
[ A 125.329486] A DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 007b
[ A 125.329493] Code: 18 1f 14 c2 85 ff 0f 85 41 ff ff ff f0 ff 05 38 fb 02 c2 e9 35 ff ff ff 8d 76 00 89 44 24 04 c7 04 24 55 93 f3 c1 e8 8c e7 f5 ff <0f> ff 8b 5d f4 31 c0 8b 75 f8 8b 7d fc 89 ec 5d c3 64 ff 05 18 [ A 125.329558] ---[ end trace dd2704ca649b44ba ]---

#10 lore raw
Quoted 8 lines · On Tue, Jan 30, 2018 at 11:26:42AM +1300, [email protected] wrote:

After, received this stack.

Have not tried memtest86. These are production hosts. This has occurred on multiple hosts. I can only recall this occurring on 32 bit kernels. I cannot recall issues with other VMs not running that kernel on the same hosts.

[  125.329163] Bad swp_entry: e000000

Mixed news here then ... 'e' is 8 | 4 | 2, so it's not a single bitflip. So no point in running memtest86.

I should have made the printk produce leading zeroes, because that's 0x0e00'0000. ptes use the top 5 bits to encode the swapfile, so this swap entry is decoded as swapfile 1, page number 0x0600'0000. That's clearly ludicrous because you don't have a swapfile 1, and if you did, it wouldn't be so large as a terabyte.

I think the next step in debugging this is printing the PTE which gave us this swp_entry. If you can drop the patch I asked you to try, and apply this patch instead, we'll have more idea about what's going on.

Thanks!

mm/memory.c +4 -0
index 403934297a3d..8caaddb07747 100644
@@ -2892,6 +2892,10 @@ int do_swap_page(struct vm_fault *vmf)
 	if (!page)
 		page = lookup_swap_cache(entry, vma_readahead ? vma : NULL,
 					 vmf->address);
+	if (IS_ERR(page)) {
+		pte_ERROR(vmf->orig_pte);
+		page = NULL;
+	}
 	if (!page) {
 		struct swap_info_struct *si = swp_swap_info(entry);
 
mm/shmem.c +4 -0
index 7fbe67be86fa..905fa34e022a 100644
@@ -1651,6 +1651,10 @@ static int shmem_getpage_gfp(struct inode *inode, pgoff_t index,
 	if (swap.val) {
 		/* Look it up and read it in.. */
 		page = lookup_swap_cache(swap, NULL, 0);
+		if (IS_ERR(page)) {
+			pte_ERROR(vmf->orig_pte);
+			page = NULL;
+		}
 		if (!page) {
 			/* Or update major stats only when swapin succeeds?? */
 			if (fault_type) {
mm/swap_state.c +11 -1
index 39ae7cfad90f..7ee594c8eadd 100644
@@ -334,8 +334,14 @@ struct page *lookup_swap_cache(swp_entry_t entry, struct vm_area_struct *vma,
 	struct page *page;
 	unsigned long ra_info;
 	int win, hits, readahead;
+	struct address_space *swapper_space = swap_address_space(entry);
+
+	if (!swapper_space) {
+		pr_err("Bad swp_entry: %lx\n", entry.val);
+		return ERR_PTR(-EFAULT);
+	}
 
-	page = find_get_page(swap_address_space(entry), swp_offset(entry));
+	page = find_get_page(swapper_space, swp_offset(entry));
 
 	INC_CACHE_INFO(find_total);
 	if (page) {
@@ -676,6 +682,10 @@ struct page *swap_readahead_detect(struct vm_fault *vmf,
 	if ((unlikely(non_swap_entry(entry))))
 		return NULL;
 	page = lookup_swap_cache(entry, vma, faddr);
+	if (IS_ERR(page)) {
+		pte_ERROR(vmf->orig_pte);
+		page = NULL;
+	}
 	if (page)
 		return page;
 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#11 lore raw

https://bugzilla.redhat.com/show_bug.cgi?id=1531779

It might be something related that
"x86/mm: Found insecure W+X mapping at address" message is printed at boot.

Are you seeing "x86/mm: Found insecure W+X mapping at address" before hitting "BUG: unable to handle kernel NULL pointer dereference" ?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#12 lore raw
Quoted 7 lines · Tetsuo Handa

https://bugzilla.redhat.com/show_bug.cgi?id=1531779

It might be something related that
"x86/mm: Found insecure W+X mapping at address" message is printed at boot.

Are you seeing "x86/mm: Found insecure W+X mapping at address" before hitting "BUG: unable to handle kernel NULL pointer dereference" ?

There are about eight different bugs in that thread; the only commonality I see between them is that there's a null pointer dereference somewhere in the kernel.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#13 lore raw
In reply to Matthew Wilcox

ping?

Quoted 88 lines · Matthew Wilcox
Quoted 8 lines · On Tue, Jan 30, 2018 at 11:26:42AM +1300, [email protected] wrote:

After, received this stack.

Have not tried memtest86. These are production hosts. This has occurred on multiple hosts. I can only recall this occurring on 32 bit kernels. I cannot recall issues with other VMs not running that kernel on the same hosts.

[  125.329163] Bad swp_entry: e000000

Mixed news here then ... 'e' is 8 | 4 | 2, so it's not a single bitflip. So no point in running memtest86.

I should have made the printk produce leading zeroes, because that's 0x0e00'0000. ptes use the top 5 bits to encode the swapfile, so this swap entry is decoded as swapfile 1, page number 0x0600'0000. That's clearly ludicrous because you don't have a swapfile 1, and if you did, it wouldn't be so large as a terabyte.

I think the next step in debugging this is printing the PTE which gave us this swp_entry. If you can drop the patch I asked you to try, and apply this patch instead, we'll have more idea about what's going on.

Thanks!

mm/memory.c +4 -0
index 403934297a3d..8caaddb07747 100644
@@ -2892,6 +2892,10 @@ int do_swap_page(struct vm_fault *vmf)
 	if (!page)
 		page = lookup_swap_cache(entry, vma_readahead ? vma : NULL,
 					 vmf->address);
+	if (IS_ERR(page)) {
+		pte_ERROR(vmf->orig_pte);
+		page = NULL;
+	}
 	if (!page) {
 		struct swap_info_struct *si = swp_swap_info(entry);
 
mm/shmem.c +4 -0
index 7fbe67be86fa..905fa34e022a 100644
@@ -1651,6 +1651,10 @@ static int shmem_getpage_gfp(struct inode *inode, pgoff_t index,
 	if (swap.val) {
 		/* Look it up and read it in.. */
 		page = lookup_swap_cache(swap, NULL, 0);
+		if (IS_ERR(page)) {
+			pte_ERROR(vmf->orig_pte);
+			page = NULL;
+		}
 		if (!page) {
 			/* Or update major stats only when swapin succeeds?? */
 			if (fault_type) {
mm/swap_state.c +11 -2
index 39ae7cfad90f..7ee594c8eadd 100644
@@ -334,8 +334,14 @@ struct page *lookup_swap_cache(swp_entry_t entry, struct vm_area_struct *vma,
 	struct page *page;
 	unsigned long ra_info;
 	int win, hits, readahead;
+	struct address_space *swapper_space = swap_address_space(entry);
+
+	if (!swapper_space) {
+		pr_err("Bad swp_entry: %lx\n", entry.val);
+		return ERR_PTR(-EFAULT);
+	}
 
-	page = find_get_page(swap_address_space(entry), swp_offset(entry));
+	page = find_get_page(swapper_space, swp_offset(entry));
 
 	INC_CACHE_INFO(find_total);
 	if (page) {
@@ -676,6 +682,10 @@ struct page *swap_readahead_detect(struct vm_fault *vmf,
 	if ((unlikely(non_swap_entry(entry))))
 		return NULL;
 	page = lookup_swap_cache(entry, vma, faddr);
+	if (IS_ERR(page)) {
+		pte_ERROR(vmf->orig_pte);
+		page = NULL;
+	}
 	if (page)
 		return page;
 
 
--
Quoted 4 lines

To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

author maintainer
#14 lore raw
Quoted 3 lines · On Fri, 9 Feb 2018 06:47:26 -0800 Matthew Wilcox <[email protected]> wrote:

ping?

There have been a bunch of updates to this issue in bugzilla
(https://bugzilla.kernel.org/show_bug.cgi?id=198497). Sigh, I don't know what to do about this - maybe there's some way of getting bugzilla to echo everything to linux-mm or something.

Anyway, please take a look - we appear to have a bug here. Perhaps this bug is sufficiently gnarly for you to prepare a debugging patch which we can add to the mainline kernel so we get (much) more debugging info when people hit it?

#15 lore raw
Quoted 11 lines · Andrew Morton
Quoted 3 lines · On Fri, 9 Feb 2018 06:47:26 -0800 Matthew Wilcox <[email protected]> wrote:

ping?

There have been a bunch of updates to this issue in bugzilla
(https://bugzilla.kernel.org/show_bug.cgi?id=198497). Sigh, I don't know what to do about this - maybe there's some way of getting bugzilla to echo everything to linux-mm or something.

Quoted 4 lines

Anyway, please take a look - we appear to have a bug here. Perhaps this bug is sufficiently gnarly for you to prepare a debugging patch which we can add to the mainline kernel so we get (much) more debugging info when people hit it?

I have a few thoughts ...

- The debugging patch I prepared appears to be doing its job well. People get the message and their machine stays working.
- The commonality appears to be Xen running 32-bit kernels. Maybe we can kick the problem over to them to solve?
- If we are seeing corruption purely in the lower bits, *we'll never know*. The radix tree lookup will simply not find anything, and all will be well. That said, the bad PTE values reported in that bug have the NX bit and one other bit set; generally bit 32, 33 or 34. I have an idea for adding a parity bit, but haven't had time to implement it. Anyone have an intern who wants an interesting kernel project to work on?

Given that this is happening on Xen, I wonder if Xen is using some of the bits in the page table for its own purposes.