Skip to contentSkip to conversation

pv_ops - 2.6.26 - unable to handle kernel paging request

DISCUSSION Christopher S. Aker 2 participants 4 messages Jul 22, 2008 virtio core

lore
Share & tools
lore
#0 lore raw

Xen: 3.1.2 (or thereabouts), 64bit
dom0: 2.6.18.8, pae
pv-ops, 2.6.26

BUG: unable to handle kernel paging request at 69746174
IP: [<c015e221>] move_freepages+0x61/0xc0
*pdpt = 0000000204ed6007
Oops: 0002 [#1] SMP
Modules linked in:

Pid: 6859, comm: sh Not tainted (2.6.26-linode13 #1)
EIP: 0061:[<c015e221>] EFLAGS: 00010002 CPU: 2
EIP is at move_freepages+0x61/0xc0
EAX: 69746174 EBX: 25413325 ECX: c158e038 EDX: 732e316d
ESI: c158e020 EDI: 00000000 EBP: c158ffe0 ESP: ec2cddf8
DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0069
Process sh (pid: 6859, ti=ec2cc000 task=ecd3f400 task.ti=ec2cc000)

Stack: c0630200 00000008 0002c7ff c1588000 c0630200 c158ffe0 c015e2ea 
00000001
00000001 00000001 c158f6e0 00000000 c0630200 c015e5d9 c0630a84 00000000
c0630a84 00000000 00000008 00000000 c1587418 c0630200 00000018 0000001f
Call Trace:
[<c015e2ea>] move_freepages_block+0x6a/0x80
[<c015e5d9>] __rmqueue+0x1a9/0x1e0
[<c015e651>] rmqueue_bulk+0x41/0x70
[<c015eae4>] get_page_from_freelist+0x464/0x490
[<c015ebba>] __alloc_pages_internal+0xaa/0x460
[<c015ef8f>] __alloc_pages+0xf/0x20
[<c015f4bf>] __get_free_pages+0xf/0x20
[<c01c015f>] proc_file_read+0x8f/0x2a0
[<c01c00d0>] proc_file_read+0x0/0x2a0
[<c01bb7ca>] proc_reg_read+0x5a/0x90
[<c01801f1>] vfs_read+0xa1/0x160
[<c01bb770>] proc_reg_read+0x0/0x90
[<c0180551>] sys_read+0x41/0x70
[<c0107256>] syscall_call+0x7/0xb
=======================
Code: cb 77 6f 8b 44 24 1c 89 de c1 e0 03 89 44 24 04 eb 07 83 c6 20 39 
f5 72 59 f6 46 02 04 74 f3 8d 4e 18 8b 56 18 8b 41 04 8b 5e 0c <89> 10 
89 42 04 8d 04 9b c7 46 18 00 01 10 00 8d 04 43 8b 14 24
EIP: [<c015e221>] move_freepages+0x61/0xc0 SS:ESP 0069:ec2cddf8
---[ end trace 628f7b31d5a52105 ]---

Kernel binary is located here:

http://www.theshore.net/~caker/kernels/2.6.26-linode13

-Chris

#1 lore raw
Quoted 3 lines · Christopher S. Aker

Xen: 3.1.2 (or thereabouts), 64bit
dom0: 2.6.18.8, pae
pv-ops, 2.6.26

Just as I'm about to announce something like "no known bugs", you pop up with one ;)

Quoted 39 lines
BUG: unable to handle kernel paging request at 69746174
IP: [<c015e221>] move_freepages+0x61/0xc0
*pdpt = 0000000204ed6007
Oops: 0002 [#1] SMP
Modules linked in:

Pid: 6859, comm: sh Not tainted (2.6.26-linode13 #1)
EIP: 0061:[<c015e221>] EFLAGS: 00010002 CPU: 2
EIP is at move_freepages+0x61/0xc0
EAX: 69746174 EBX: 25413325 ECX: c158e038 EDX: 732e316d
ESI: c158e020 EDI: 00000000 EBP: c158ffe0 ESP: ec2cddf8
DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0069
Process sh (pid: 6859, ti=ec2cc000 task=ecd3f400 task.ti=ec2cc000)

Stack: c0630200 00000008 0002c7ff c1588000 c0630200 c158ffe0 c015e2ea 
00000001
00000001 00000001 c158f6e0 00000000 c0630200 c015e5d9 c0630a84 00000000
c0630a84 00000000 00000008 00000000 c1587418 c0630200 00000018 0000001f
Call Trace:
[<c015e2ea>] move_freepages_block+0x6a/0x80
[<c015e5d9>] __rmqueue+0x1a9/0x1e0
[<c015e651>] rmqueue_bulk+0x41/0x70
[<c015eae4>] get_page_from_freelist+0x464/0x490
[<c015ebba>] __alloc_pages_internal+0xaa/0x460
[<c015ef8f>] __alloc_pages+0xf/0x20
[<c015f4bf>] __get_free_pages+0xf/0x20
[<c01c015f>] proc_file_read+0x8f/0x2a0
[<c01c00d0>] proc_file_read+0x0/0x2a0
[<c01bb7ca>] proc_reg_read+0x5a/0x90
[<c01801f1>] vfs_read+0xa1/0x160
[<c01bb770>] proc_reg_read+0x0/0x90
[<c0180551>] sys_read+0x41/0x70
[<c0107256>] syscall_call+0x7/0xb
=======================
Code: cb 77 6f 8b 44 24 1c 89 de c1 e0 03 89 44 24 04 eb 07 83 c6 20 
39 f5 72 59 f6 46 02 04 74 f3 8d 4e 18 8b 56 18 8b 41 04 8b 5e 0c <89> 
10 89 42 04 8d 04 9b c7 46 18 00 01 10 00 8d 04 43 8b 14 24
EIP: [<c015e221>] move_freepages+0x61/0xc0 SS:ESP 0069:ec2cddf8
---[ end trace 628f7b31d5a52105 ]---
Quoted 3 lines

Kernel binary is located here:

http://www.theshore.net/~caker/kernels/2.6.26-linode13

Thanks.

What was going on at the time? Was the system idle? Under load? Does it happen during boot, or after some uptime? (Pid 6859 suggests the system has been up for a while.)

J

#2 lore raw
In reply to Christopher S. Aker
Quoted 3 lines · Christopher S. Aker

Xen: 3.1.2 (or thereabouts), 64bit
dom0: 2.6.18.8, pae
pv-ops, 2.6.26

What's the .config for this kernel? Do you know what /proc file it's trying to access at the time?

Quoted 1 line
BUG: unable to handle kernel paging request at 69746174

This is address is ascii "tati". Likely to be use-after-free, though it could be the result of a wild write.

The code seems to correspond to the line:

		list_add(&page->lru,
			&zone->free_area[order].free_list[migratetype]);

so it suggests that either the zone freelist or the page structure is corrupted.

Quoted 9 lines

IP: [<c015e221>] move_freepages+0x61/0xc0
*pdpt = 0000000204ed6007

Oops: 0002 [#1] SMP
Modules linked in:

Pid: 6859, comm: sh Not tainted (2.6.26-linode13 #1)
EIP: 0061:[<c015e221>] EFLAGS: 00010002 CPU: 2 EIP is at move_freepages+0x61/0xc0
EAX: 69746174 EBX: 25413325 ECX: c158e038 EDX: 732e316d

EBX="%31%"
EDX="m1.~"

EAX, EBX and EDX are all loaded from the page structure, so it's definitely been hit with something. Or perhaps the page pointer was wrong in the first place. If page_order() gets corrupted for the page, then it could cause that loop to march off into nowhere.

Could you try again with DEBUG_PAGEALLOC turned on?

Thanks,
J

Quoted 34 lines

ESI: c158e020 EDI: 00000000 EBP: c158ffe0 ESP: ec2cddf8
DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0069
Process sh (pid: 6859, ti=ec2cc000 task=ecd3f400 task.ti=ec2cc000)

Stack: c0630200 00000008 0002c7ff c1588000 c0630200 c158ffe0 c015e2ea 
00000001
00000001 00000001 c158f6e0 00000000 c0630200 c015e5d9 c0630a84 00000000
c0630a84 00000000 00000008 00000000 c1587418 c0630200 00000018 0000001f
Call Trace:
[<c015e2ea>] move_freepages_block+0x6a/0x80
[<c015e5d9>] __rmqueue+0x1a9/0x1e0
[<c015e651>] rmqueue_bulk+0x41/0x70
[<c015eae4>] get_page_from_freelist+0x464/0x490
[<c015ebba>] __alloc_pages_internal+0xaa/0x460
[<c015ef8f>] __alloc_pages+0xf/0x20
[<c015f4bf>] __get_free_pages+0xf/0x20
[<c01c015f>] proc_file_read+0x8f/0x2a0
[<c01c00d0>] proc_file_read+0x0/0x2a0
[<c01bb7ca>] proc_reg_read+0x5a/0x90
[<c01801f1>] vfs_read+0xa1/0x160
[<c01bb770>] proc_reg_read+0x0/0x90
[<c0180551>] sys_read+0x41/0x70
[<c0107256>] syscall_call+0x7/0xb
=======================
Code: cb 77 6f 8b 44 24 1c 89 de c1 e0 03 89 44 24 04 eb 07 83 c6 20 
39 f5 72 59 f6 46 02 04 74 f3 8d 4e 18 8b 56 18 8b 41 04 8b 5e 0c <89> 
10 89 42 04 8d 04 9b c7 46 18 00 01 10 00 8d 04 43 8b 14 24
EIP: [<c015e221>] move_freepages+0x61/0xc0 SS:ESP 0069:ec2cddf8
---[ end trace 628f7b31d5a52105 ]---

Kernel binary is located here:

http://www.theshore.net/~caker/kernels/2.6.26-linode13

-Chris

#3 lore raw
In reply to Christopher S. Aker
Quoted 3 lines · Christopher S. Aker

Xen: 3.1.2 (or thereabouts), 64bit
dom0: 2.6.18.8, pae
pv-ops, 2.6.26

What's the .config for this kernel? Do you know what /proc file it's trying to access at the time?

Quoted 1 line
BUG: unable to handle kernel paging request at 69746174

This is address is ascii "tati". Likely to be use-after-free, though it could be the result of a wild write.

The code seems to correspond to the line:

		list_add(&page->lru,
			&zone->free_area[order].free_list[migratetype]);

so it suggests that either the zone freelist or the page structure is corrupted.

Quoted 9 lines

IP: [<c015e221>] move_freepages+0x61/0xc0
*pdpt = 0000000204ed6007

Oops: 0002 [#1] SMP
Modules linked in:

Pid: 6859, comm: sh Not tainted (2.6.26-linode13 #1)
EIP: 0061:[<c015e221>] EFLAGS: 00010002 CPU: 2 EIP is at move_freepages+0x61/0xc0
EAX: 69746174 EBX: 25413325 ECX: c158e038 EDX: 732e316d

EBX="%31%"
EDX="m1.~"

EAX, EBX and EDX are all loaded from the page structure, so it's definitely been hit with something. Or perhaps the page pointer was wrong in the first place. If page_order() gets corrupted for the page, then it could cause that loop to march off into nowhere.

Could you try again with DEBUG_PAGEALLOC turned on?

Thanks,
J

Quoted 31 lines

ESI: c158e020 EDI: 00000000 EBP: c158ffe0 ESP: ec2cddf8
DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0069
Process sh (pid: 6859, ti=ec2cc000 task=ecd3f400 task.ti=ec2cc000)

Stack: c0630200 00000008 0002c7ff c1588000 c0630200 c158ffe0 c015e2ea 00000001
00000001 00000001 c158f6e0 00000000 c0630200 c015e5d9 c0630a84 00000000
c0630a84 00000000 00000008 00000000 c1587418 c0630200 00000018 0000001f
Call Trace:
[<c015e2ea>] move_freepages_block+0x6a/0x80
[<c015e5d9>] __rmqueue+0x1a9/0x1e0
[<c015e651>] rmqueue_bulk+0x41/0x70
[<c015eae4>] get_page_from_freelist+0x464/0x490
[<c015ebba>] __alloc_pages_internal+0xaa/0x460
[<c015ef8f>] __alloc_pages+0xf/0x20
[<c015f4bf>] __get_free_pages+0xf/0x20
[<c01c015f>] proc_file_read+0x8f/0x2a0
[<c01c00d0>] proc_file_read+0x0/0x2a0
[<c01bb7ca>] proc_reg_read+0x5a/0x90
[<c01801f1>] vfs_read+0xa1/0x160
[<c01bb770>] proc_reg_read+0x0/0x90
[<c0180551>] sys_read+0x41/0x70
[<c0107256>] syscall_call+0x7/0xb
=======================
Code: cb 77 6f 8b 44 24 1c 89 de c1 e0 03 89 44 24 04 eb 07 83 c6 20 39 f5 72 59 f6 46 02 04 74 f3 8d 4e 18 8b 56 18 8b 41 04 8b 5e 0c <89> 10 89 42 04 8d 04 9b c7 46 18 00 01 10 00 8d 04 43 8b 14 24
EIP: [<c015e221>] move_freepages+0x61/0xc0 SS:ESP 0069:ec2cddf8
---[ end trace 628f7b31d5a52105 ]---

Kernel binary is located here:

http://www.theshore.net/~caker/kernels/2.6.26-linode13

-Chris