Invalid operand: kernel BUG at mm/rmap.c:434! and arch/i386/mm/highmem.c:42!)
I'm running kernel 2.6.9-22.ELsmp on dual Xeon servers. I've received kernel panics occasionally in the past, but they are more frequent now as the load on the system has increased. Below is a capture of the kernel panic.
If anything below screams it's coming from a certain source (defective RAM? Bad application or device driver?), please let me know as I'm pulling what little hair I have left out on this one. Suggestions on what to try would be greatly appreciated. Thanks!
------------[ cut here ]------------ kernel BUG at mm/rmap.c:434! invalid operand: 0000 [#1] SMP Modules linked in: fusedriver(U) md5 ipv6 parport_pc
lp parport autofs4 i2c_dev i2c_core sunrpc dm_multipath button battery ac uhci_hcd ehci_hcd e1000 floppy dm_snapshot dm_zero dm_mirror ext3 jbd dm_mod 3w_9xxx(U) sd_mod scsi_mod
CPU: 5 EIP: 0060:[<c01518c6>] Not tainted VLI EFLAGS: 00010202 (2.6.9-22.ELsmp)
EIP is at page_add_anon_rmap+0xe/0x66
eax: 40000964 ebx: c1800040 ecx: 090a100c edx:e5b5a544
esi: d1837e8c edi: fff25508 ebp: c1800040 esp:cf12ae40
ds: 007b es: 007b ss: 0068
Process check-enqueued- (pid: 17602,
threadinfo=cf12a000 task=f57577b0)
Stack: 40002067 80000000 c014d034 fff29000 40002025
80000000 00000163 e5b5a544
f35b9080 00000000 fff25508 fff25508 090a100c
c014d0e2 cc5a2240 00000001
090a100c ffffffff ffffffff 00000000 00000000
80000000 00000000 00000000
Call Trace:
[<c014d034>] do_anonymous_page+0x19c/0x1db
[<c014d0e2>] do_no_page+0x6f/0x2f9
[<c014d503>] handle_mm_fault+0xbd/0x175
[<c011addb>] do_page_fault+0x1ae/0x5c6
[<c014e58e>] vma_adjust+0x286/0x2d6
[<c014e762>] vma_merge+0xe1/0x165
[<c011d3d5>] finish_task_switch+0x30/0x66
[<c02cf368>] schedule+0x844/0x87a
[<c011ac2d>] do_page_fault+0x0/0x5c6
[<c02d1bab>] error_code+0x2f/0x38
Code: 83 7b 10 00 74 0b 89 ca 89 d8 e8 fb fe ff ff 01c6 89 d8 e8 ac df fe ff 5b 89 f0 5e c3 56 53 89 c3 8b 00 8b 72 44 f6 c4 08 74 08 <0f> 0b b2 01 ce 52 2e c0 85 f6 75 08 0f 0b b3 01 ce 52 2e c0 8b
<0>Fatal exception: panic in 5 seconds
bad: scheduling while atomic!
[<c02ceb51>] schedule+0x2d/0x87a [<c020a293>] poke_blanked_console+0x3d/0x9a [<c02096a5>] vt_console_print+0x294/0x2a5 [<c0129715>] __mod_timer+0x101/0x10b [<c02cf8a0>] schedule_timeout+0xd3/0xee [<c0129fba>] process_timeout+0x0/0x5 [<c0122320>] printk+0xe/0x11 [<c0106092>] die+0x15a/0x16b [<c01063f5>] do_invalid_op+0xcf/0xf2 [<f88aa3d6>] ext3_get_inode_loc+0x4f/0x226 [ext3] [<c01518c6>] page_add_anon_rmap+0xe/0x66 [<c011e5b5>] __wake_up+0x29/0x3c [<c0142c1b>] __rmqueue+0xc1/0x10c [<c0142cc1>] rmqueue_bulk+0x5b/0x65 [<c0106326>] do_invalid_op+0x0/0xf2 [<c02d1bab>] error_code+0x2f/0x38 [<c01518c6>] page_add_anon_rmap+0xe/0x66 [<c014d034>] do_anonymous_page+0x19c/0x1db [<c014d0e2>] do_no_page+0x6f/0x2f9 [<c014d503>] handle_mm_fault+0xbd/0x175 [<c011addb>] do_page_fault+0x1ae/0x5c6 [<c014e58e>] vma_adjust+0x286/0x2d6 [<c014e762>] vma_merge+0xe1/0x165 [<c011d3d5>] finish_task_switch+0x30/0x66 [<c02cf368>] schedule+0x844/0x87a [<c011ac2d>] do_page_fault+0x0/0x5c6 [<c02d1bab>] error_code+0x2f/0x38 ------------[ cut here ]------------ kernel BUG at arch/i386/mm/highmem.c:42! invalid operand: 0000 [#2] SMP Modules linked in: fusedriver(U) md5 ipv6 parport_pc
lp parport autofs4 i2c_dev i2c_core sunrpc dm_multipath button battery ac uhci_hcd ehci_hcd e1000 floppy dm_snapshot dm_zero dm_mirror ext3 jbd dm_mod 3w_9xxx(U) sd_mod scsi_mod
CPU: 5 EIP: 0060:[<c011c6e6>] Not tainted VLI EFLAGS: 00010286 (2.6.9-22.ELsmp)
EIP is at kmap_atomic+0x73/0x178
eax: c000a928 ebx: 80000000 ecx: 90b52163 edx:000000c4
esi: f467a380 edi: d898bbe8 ebp: c000af48 esp:eee66e7c
ds: 007b es: 007b ss: 0068
Process search_pe (pid: 17511, threadinfo=eee66000 task=f511e030)
Stack: 00000001 00000000 f5646900 00000001 00000000
00000000 00000000 00000000
00000000 fff25000 c182d280 e19cb630 f467a380
d898bbe8 00000c00 c014b006
e19cb630 e19cb630 e19cb630 f467a3b0 d898bbe8
afb80004 c014d4d1 f3aae22c
Call Trace:
[<c014b006>] pte_alloc_map+0xd9/0xe2
[<c014d4d1>] handle_mm_fault+0x8b/0x175
[<c011addb>] do_page_fault+0x1ae/0x5c6
[<c014e7d7>] vma_merge+0x156/0x165
[<c014ec91>] do_mmap_pgoff+0x3cd/0x666
[<c014ee2c>] do_mmap_pgoff+0x568/0x666
[<c010b67f>] sys_mmap2+0x7e/0xaf
[<c011ac2d>] do_page_fault+0x0/0x5c6
[<c02d1bab>] error_code+0x2f/0x38
[<c02d007b>] __lock_text_end+0x11a/0x100f
Code: c8 40 c0 01 c2 8d 42 16 c1 e0 0c 29 c1 89 4c 2424 8d 04 d5 00 00 00 00 89 e9 29 c1 89 c8 8b 09 8b 58 04 85 c9 75 04 85 db 74 08 <0f> 0b 2a 00 04 27 2e c0 8b 5c 24 28 8b 0d 78 29 32 c0 8b 03 89
<0>Fatal exception: panic in 5 seconds
bad: scheduling while atomic!
[<c02ceb51>] schedule+0x2d/0x87a [<c020a293>] poke_blanked_console+0x3d/0x9a [<c02096a5>] vt_console_print+0x294/0x2a5 [<c0129715>] __mod_timer+0x101/0x10b [<c02cf8a0>] schedule_timeout+0xd3/0xee [<c0129fba>] process_timeout+0x0/0x5 [<c0122320>] printk+0xe/0x11 [<c0106092>] die+0x15a/0x16b [<c01063f5>] do_invalid_op+0xcf/0xf2 [<c011ffbe>] autoremove_wake_function+0xd/0x2d [<c011c6e6>] kmap_atomic+0x73/0x178 [<c011e5b5>] __wake_up+0x29/0x3c [<c0278dc3>] __kfree_skb+0xf4/0xf7 [<c02cbcad>] unix_stream_recvmsg+0x2cc/0x39d [<c0106326>] do_invalid_op+0x0/0xf2 [<c02d1bab>] error_code+0x2f/0x38 [<c011c6e6>] kmap_atomic+0x73/0x178 [<c014b006>] pte_alloc_map+0xd9/0xe2 [<c014d4d1>] handle_mm_fault+0x8b/0x175 [<c011addb>] do_page_fault+0x1ae/0x5c6 [<c014e7d7>] vma_merge+0x156/0x165 [<c014ec91>] do_mmap_pgoff+0x3cd/0x666 [<c014ee2c>] do_mmap_pgoff+0x568/0x666 [<c010b67f>] sys_mmap2+0x7e/0xaf [<c011ac2d>] do_page_fault+0x0/0x5c6 [<c02d1bab>] error_code+0x2f/0x38 [<c02d007b>] __lock_text_end+0x11a/0x100f ------------[ cut here ]------------ kernel BUG at arch/i386/mm/highmem.c:42! invalid operand: 0000 [#3] SMP Modules linked in: fusedriver(U) md5 ipv6 parport_pc
lp parport autofs4 i2c_dev i2c_core sunrpc dm_multipath button battery ac uhci_hcd ehci_hcd e1000 floppy dm_snapshot dm_zero dm_mirror ext3 jbd dm_mod 3w_9xxx(U) sd_mod scsi_mod
CPU: 5 EIP: 0060:[<c011c6e6>] Not tainted VLI EFLAGS: 00010286 (2.6.9-22.ELsmp)
EIP is at kmap_atomic+0x73/0x178
eax: c000a928 ebx: 80000000 ecx: 90b52163 edx:000000c4
esi: f5634080 edi: f4c3ddf8 ebp: c000af48 esp:f560edd0
ds: 007b es: 007b ss: 0068
Process irqbalance (pid: 2543, threadinfo=f560e000 task=f4ec8330)
Stack: f7ee1e00 c3802ab8 00000000 e0f6a4b4 f778f478
e0f6a474 e0f6a474 00000000
00000000 fff25000 c35d96c0 f7f10ef0 f5634080
f4c3ddf8 00000ff8 c014b006
f7f10ef0 f7f10ef0 f7f10ef0 f56340b0 f4c3ddf8
b7fff000 c014d4d1 f54de90c
Call Trace:
[<c014b006>] pte_alloc_map+0xd9/0xe2
[<c014d4d1>] handle_mm_fault+0x8b/0x175
[<c011addb>] do_page_fault+0x1ae/0x5c6
[<c016dbc8>] dput+0x34/0x1a7
[<c0166050>] link_path_walk+0x94/0xbe
[<c02cf8a7>] schedule_timeout+0xda/0xee
[<c01c0f8d>] vsnprintf+0x448/0x488
[<c011ac2d>] do_page_fault+0x0/0x5c6
[<c02d1bab>] error_code+0x2f/0x38
[<c01c1c8a>] __copy_to_user_ll+0x30/0x46
[<c0173e13>] seq_read+0x24a/0x2c2
[<c0159c61>] vfs_read+0xb6/0xe2
[<c0159e74>] sys_read+0x3c/0x62
[<c02d10cf>] syscall_call+0x7/0xb
Code: c8 40 c0 01 c2 8d 42 16 c1 e0 0c 29 c1 89 4c 2424 8d 04 d5 00 00 00 00 89 e9 29 c1 89 c8 8b 09 8b 58 04 85 c9 75 04 85 db 74 08 <0f> 0b 2a 00 04 27 2e c0 8b 5c 24 28 8b 0d 78 29 32 c0 8b 03 89
<0>Fatal exception: panic in 5 seconds
bad: scheduling while atomic!
[<c02ceb51>] schedule+0x2d/0x87a [<c020a293>] poke_blanked_console+0x3d/0x9a [<c02096a5>] vt_console_print+0x294/0x2a5 [<c0129715>] __mod_timer+0x101/0x10b [<c02cf8a0>] schedule_timeout+0xd3/0xee [<c0129fba>] process_timeout+0x0/0x5 [<c0122320>] printk+0xe/0x11 [<c0106092>] die+0x15a/0x16b [<c01063f5>] do_invalid_op+0xcf/0xf2 [<c0142c1b>] __rmqueue+0xc1/0x10c [<c011c6e6>] kmap_atomic+0x73/0x178 [<c0142cc1>] rmqueue_bulk+0x5b/0x65 [<c0143050>] buffered_rmqueue+0x17d/0x1a5 [<c0106326>] do_invalid_op+0x0/0xf2 [<c02d1bab>] error_code+0x2f/0x38 [<c011c6e6>] kmap_atomic+0x73/0x178 [<c014b006>] pte_alloc_map+0xd9/0xe2 [<c014d4d1>] handle_mm_fault+0x8b/0x175 [<c011addb>] do_page_fault+0x1ae/0x5c6 [<c016dbc8>] dput+0x34/0x1a7 [<c0166050>] link_path_walk+0x94/0xbe [<c02cf8a7>] schedule_timeout+0xda/0xee [<c01c0f8d>] vsnprintf+0x448/0x488 [<c011ac2d>] do_page_fault+0x0/0x5c6 [<c02d1bab>] error_code+0x2f/0x38 [<c01c1c8a>] __copy_to_user_ll+0x30/0x46 [<c0173e13>] seq_read+0x24a/0x2c2 [<c0159c61>] vfs_read+0xb6/0xe2 [<c0159e74>] sys_read+0x3c/0x62 [<c02d10cf>] syscall_call+0x7/0xb ------------[ cut here ]------------ kernel BUG at arch/i386/mm/highmem.c:42! invalid operand: 0000 [#4] SMP Modules linked in: fusedriver(U) md5 ipv6 parport_pc
lp parport autofs4 i2c_dev i2c_core sunrpc dm_multipath button battery ac uhci_hcd ehci_hcd e1000 floppy dm_snapshot dm_zero dm_mirror ext3 jbd dm_mod 3w_9xxx(U) sd_mod scsi_mod
CPU: 5 EIP: 0060:[<c011c6e6>] Not tainted VLI EFLAGS: 00010286 (2.6.9-22.ELsmp)
EIP is at kmap_atomic+0x73/0x178
eax: c000a928 ebx: 80000000 ecx: 90b52163 edx:000000c4
esi: f56a5980 edi: c847cbe8 ebp: c000af48 esp:d8236e7c
ds: 007b es: 007b ss: 0068
Process search_pe (pid: 17600, threadinfo=d8236000 task=f3a95930)
Stack: 00000001 00000000 ce89ea80 00000001 00000000
00000000 00000000 00000000
00000000 fff25000 c3202bc0 f7f10b30 f56a5980
c847cbe8 00000c00 c014b006
f7f10b30 f7f10b30 f7f10b30 f56a59b0 c847cbe8
afb80004 c014d4d1 f3a68a6c
Call Trace:
[<c014b006>] pte_alloc_map+0xd9/0xe2
[<c014d4d1>] handle_mm_fault+0x8b/0x175
[<c011addb>] do_page_fault+0x1ae/0x5c6
[<c014e7d7>] vma_merge+0x156/0x165
[<c014ec91>] do_mmap_pgoff+0x3cd/0x666
[<c014ee2c>] do_mmap_pgoff+0x568/0x666
[<c010b67f>] sys_mmap2+0x7e/0xaf
[<c011ac2d>] do_page_fault+0x0/0x5c6
[<c02d1bab>] error_code+0x2f/0x38
[<c02d007b>] __lock_text_end+0x11a/0x100f
Code: c8 40 c0 01 c2 8d 42 16 c1 e0 0c 29 c1 89 4c 2424 8d 04 d5 00 00 00 00 89 e9 29 c1 89 c8 8b 09 8b 58 04 85 c9 75 04 85 db 74 08 <0f> 0b 2a 00 04 27 2e c0 8b 5c 24 28 8b 0d 78 29 32 c0 8b 03 89
<0>Fatal exception: panic in 5 seconds
bad: scheduling while atomic!
[<c02ceb51>] schedule+0x2d/0x87a [<c020a293>] poke_blanked_console+0x3d/0x9a [<c02096a5>] vt_console_print+0x294/0x2a5 [<c0129715>] __mod_timer+0x101/0x10b [<c02cf8a0>] schedule_timeout+0xd3/0xee [<c0129fba>] process_timeout+0x0/0x5 [<c0122320>] printk+0xe/0x11 [<c0106092>] die+0x15a/0x16b [<c01063f5>] do_invalid_op+0xcf/0xf2 [<c011ffbe>] autoremove_wake_function+0xd/0x2d [<c011c6e6>] kmap_atomic+0x73/0x178 [<c011e5b5>] __wake_up+0x29/0x3c [<c0278dc3>] __kfree_skb+0xf4/0xf7 [<c02cbcad>] unix_stream_recvmsg+0x2cc/0x39d [<c0106326>] do_invalid_op+0x0/0xf2 [<c02d1bab>] error_code+0x2f/0x38 [<c011c6e6>] kmap_atomic+0x73/0x178 [<c014b006>] pte_alloc_map+0xd9/0xe2 [<c014d4d1>] handle_mm_fault+0x8b/0x175 [<c011addb>] do_page_fault+0x1ae/0x5c6 [<c014e7d7>] vma_merge+0x156/0x165 [<c014ec91>] do_mmap_pgoff+0x3cd/0x666 [<c014ee2c>] do_mmap_pgoff+0x568/0x666 [<c010b67f>] sys_mmap2+0x7e/0xaf [<c011ac2d>] do_page_fault+0x0/0x5c6 [<c02d1bab>] error_code+0x2f/0x38 [<c02d007b>] __lock_text_end+0x11a/0x100f ------------[ cut here ]------------ kernel BUG at arch/i386/mm/highmem.c:42! invalid operand: 0000 [#5] SMP Modules linked in: fusedriver(U) md5 ipv6 parport_pc
lp parport autofs4 i2c_dev i2c_core sunrpc dm_multipath button battery ac uhci_hcd ehci_hcd e1000 floppy dm_snapshot dm_zero dm_mirror ext3 jbd dm_mod 3w_9xxx(U) sd_mod scsi_mod
CPU: 5 EIP: 0060:[<c011c6e6>] Not tainted VLI EFLAGS: 00010286 (2.6.9-22.ELsmp)
EIP is at kmap_atomic+0x73/0x178
eax: c000a928 ebx: 80000000 ecx: 90b52163 edx:000000c4
esi: f4705380 edi: f4d065a8 ebp: c000af48 esp:f55eee7c
ds: 007b es: 007b ss: 0068
Process mysqld (pid: 2882, threadinfo=f55ee000 task=f56c7230)
Stack: 00000057 00000000 f5754b80 00000057 00000001
c0140263 00000000 00000000
00000000 fff25000 c295d2a0 f7f10550 f4705380
f4d065a8 000002f8 c014b006
f7f10550 f7f10550 f7f10550 f47053b0 f4d065a8
96a5f004 c014d4d1 f3a8f7ac
Call Trace:
[<c0140263>] __generic_file_aio_read+0x19f/0x1bd
[<c014b006>] pte_alloc_map+0xd9/0xe2
[<c014d4d1>] handle_mm_fault+0x8b/0x175
[<c011addb>] do_page_fault+0x1ae/0x5c6
[<c014e1e9>] __vma_link+0x59/0x66
[<c014e23a>] vma_link+0x44/0xbc
[<c014ee2c>] do_mmap_pgoff+0x568/0x666
[<c010b67f>] sys_mmap2+0x7e/0xaf
[<c011ac2d>] do_page_fault+0x0/0x5c6
[<c02d1bab>] error_code+0x2f/0x38
[<c02d007b>] __lock_text_end+0x11a/0x100f
Code: c8 40 c0 01 c2 8d 42 16 c1 e0 0c 29 c1 89 4c 2424 8d 04 d5 00 00 00 00 89 e9 29 c1 89 c8 8b 09 8b 58 04 85 c9 75 04 85 db 74 08 <0f> 0b 2a 00 04 27 2e c0 8b 5c 24 28 8b 0d 78 29 32 c0 8b 03 89
<0>Fatal exception: panic in 5 seconds
bad: scheduling while atomic!
[<c02ceb51>] schedule+0x2d/0x87a [<c020a293>] poke_blanked_console+0x3d/0x9a [<c02096a5>] vt_console_print+0x294/0x2a5 [<c0129715>] __mod_timer+0x101/0x10b [<c02cf8a0>] schedule_timeout+0xd3/0xee [<c0129fba>] process_timeout+0x0/0x5 [<c0122320>] printk+0xe/0x11 [<c0106092>] die+0x15a/0x16b [<c01063f5>] do_invalid_op+0xcf/0xf2 [<c011ffbe>] autoremove_wake_function+0xd/0x2d [<c011c6e6>] kmap_atomic+0x73/0x178 [<c011e5b5>] __wake_up+0x29/0x3c [<c0278dc3>] __kfree_skb+0xf4/0xf7 [<c02cbcad>] unix_stream_recvmsg+0x2cc/0x39d [<c0106326>] do_invalid_op+0x0/0xf2 [<c02d1bab>] error_code+0x2f/0x38 [<c011c6e6>] kmap_atomic+0x73/0x178 [<c0140263>] __generic_file_aio_read+0x19f/0x1bd [<c014b006>] pte_alloc_map+0xd9/0xe2 [<c014d4d1>] handle_mm_fault+0x8b/0x175 [<c011addb>] do_page_fault+0x1ae/0x5c6 [<c014e1e9>] __vma_link+0x59/0x66 [<c014e23a>] vma_link+0x44/0xbc [<c014ee2c>] do_mmap_pgoff+0x568/0x666 [<c010b67f>] sys_mmap2+0x7e/0xaf [<c011ac2d>] do_page_fault+0x0/0x5c6 [<c02d1bab>] error_code+0x2f/0x38 [<c02d007b>] __lock_text_end+0x11a/0x100f Kernel panic - not syncing: Fatal exception
____________________________________________________________________________________
Never miss an email again!
Yahoo! Toolbar alerts you the instant new Mail arrives.
http://tools.search.yahoo.com/toolbar/features/mail/
Quoted 1 line · Pat
I'm running kernel 2.6.9-22.ELsmp on dual Xeon
You'd do better to ask Red Hat support than here.
Quoted 4 lines
servers. I've received kernel panics occasionally in the past, but they are more frequent now as the load on the system has increased. Below is a capture of the kernel panic.
Are the initial BUGs usually of that kind - rmap.c:434?
(The subsequent scheduling-while-atomics and highmem.c:42s are not interesting, just consequences of the original BUG in an untidy place.)
Quoted 5 lines
If anything below screams it's coming from a certain source (defective RAM? Bad application or device driver?), please let me know as I'm pulling what little hair I have left out on this one. Suggestions on what to try would be greatly appreciated. Thanks!
I've never seen a BUG in page_add_anon_rmap before:
there's one in page_remove_rmap which comes up from time to time, but this is different.
The page allocator has just dished out a PageReserved page (at physical 1G+8k: interesting that it's so close to 1G), which should never happen - but there was less checking for that in 2.6.9-based kernels.
Do check your RAM (memtest86 overnight), maybe that PG_reserved bit is spurious. But if rmap.c:434 is what's happening to you again and again, I'd wonder if a driver has been allocating a high-order page, marking the constituent pages as reserved, later clearing reserved from the first constituent page, then freeing the high-order page while leaving other reserved bits behind - I believe that could sneak a PageReserved page back into the 2.6.9 allocator.
If it were a particular application triggering this, it'd still be a kernel bug to allow it to happen.
Hugh
Quoted 41 lines
------------[ cut here ]------------ kernel BUG at mm/rmap.c:434! invalid operand: 0000 [#1] SMP Modules linked in: fusedriver(U) md5 ipv6 parport_pclp parport autofs4
i2c_devi2c_coresunrpcdm_multipathbutton battery acuhci_hcdehci_hcde1000 floppydm_snapshotdm_zerodm_mirrorext3 jbddm_mod3w_9xxx(U)sd_modscsi_modCPU: 5 EIP: 0060:[<c01518c6>] Not tainted VLI EFLAGS: 00010202 (2.6.9-22.ELsmp)EIP is at
page_add_anon_rmap+0xe/0x66
eax: 40000964 ebx:c1800040ecx:090a100cedx:e5b5a544
esi:d1837e8cedi:fff25508ebp:c1800040esp:cf12ae40
ds: 007b es: 007b ss: 0068
Process check-enqueued- (pid: 17602,
threadinfo=cf12a000 task=f57577b0)Stack: 40002067 80000000 c014d034 fff29000 40002025 80000000 00000163 e5b5a544 f35b9080 00000000 fff25508 fff25508 090a100c c014d0e2 cc5a2240 00000001 090a100c ffffffff ffffffff 00000000 00000000 80000000 00000000 00000000 Call Trace: [<c014d034>] do_anonymous_page+0x19c/0x1db [<c014d0e2>] do_no_page+0x6f/0x2f9 [<c014d503>] handle_mm_fault+0xbd/0x175 [<c011addb>] do_page_fault+0x1ae/0x5c6 [<c014e58e>] vma_adjust+0x286/0x2d6 [<c014e762>] vma_merge+0xe1/0x165 [<c011d3d5>] finish_task_switch+0x30/0x66 [<c02cf368>] schedule+0x844/0x87a [<c011ac2d>] do_page_fault+0x0/0x5c6 [<c02d1bab>] error_code+0x2f/0x38 Code: 83 7b 10 00 74 0b 89 ca 89 d8 e8 fb fe ff ff 01c6 89 d8 e8 ac df fe ff 5b 89 f0 5e c3 56 53 89 c3 8b 00 8b 72 44 f6 c4 08 74 08 <0f> 0b b2 01 ce 52 2e c0 85 f6 75 08 0f 0b b3 01 ce 52 2e c0 8b
Quoted 3 lines · --- Hugh Dickins
Quoted 1 line
I'm running kernel 2.6.9-22.ELsmp on dual Xeon
You'd do better to ask Red Hat support than here.
Thanks for the suggestion. I'll give them a shot too.
Quoted 2 lines
Are the initial BUGs usually of that kind - rmap.c:434?
Yes. From the ones I've recorded so far, they all begin with BUGs on rmap.c:434.
Quoted 1 line
Do check your RAM (memtest86 overnight), maybe that
I ran memtest86+ (default test) overnight on the system and it's still going strong, so I hope that indicates the RAM is okay.
Quoted 11 lines
PG_reservedbit is spurious. But if rmap.c:434 is what's happening to you again and again, I'd wonder if a driver has been allocating a high-order page, marking the constituent pages as reserved, later clearing reserved from the first constituent page, then freeing the high-order page while leaving other reserved bits behind - I believe that could sneak a PageReserved page back into the 2.6.9 allocator.If it were a particular application triggering this, it'd still be a kernel bug to allow it to happen.
Yes, there is a specific app which seems to be related to the kernel panic.
We use a few third party drivers on the system, so my initial suspicions were on: 3w_9xxx(U) which is the RAID card driver and fusedriver(U) which is a hardware PCI card driver.
The app which is kernel panicing is the only application which makes use of the fusedriver. I'd like to confirm my suspicions on the driver. Do you have any suggestions on how I could trace the kernel panic to that specific driver?
Thanks again for your input Hugh.
Pat
Quoted 46 lines
Quoted 5 lines
------------[ cut here ]------------ kernel BUG at mm/rmap.c:434! invalid operand: 0000 [#1] SMP Modules linked in: fusedriver(U) md5 ipv6
parport_pcQuoted 2 lines
lp parport autofs4
i2c_devi2c_coresunrpcdm_multipathbutton battery acuhci_hcdehci_hcde1000
Quoted 1 line
floppy
dm_snapshotdm_zerodm_mirrorext3 jbd
dm_modQuoted 6 lines
3w_9xxx(U)
sd_modscsi_modCPU: 5 EIP: 0060:[<c01518c6>] Not tainted VLI EFLAGS: 00010202 (2.6.9-22.ELsmp)EIP is at
page_add_anon_rmap+0xe/0x66
eax: 40000964 ebx:c1800040ecx:090a100cedx:
Quoted 2 lines
e5b5a544
esi:d1837e8cedi:fff25508ebp:c1800040esp:
Quoted 5 lines
cf12ae40
ds: 007b es: 007b ss: 0068
Process check-enqueued- (pid: 17602,
threadinfo=cf12a000 task=f57577b0)Stack: 40002067 80000000 c014d034 fff2900040002025
Quoted 2 lines
80000000 00000163
e5b5a544f35b9080 00000000 fff25508 fff25508
090a100cQuoted 2 lines
c014d0e2cc5a224000000001090a100c ffffffff ffffffff 0000000000000000
Quoted 13 lines
80000000 00000000 00000000
Call Trace: [<c014d034>] do_anonymous_page+0x19c/0x1db [<c014d0e2>] do_no_page+0x6f/0x2f9 [<c014d503>] handle_mm_fault+0xbd/0x175 [<c011addb>] do_page_fault+0x1ae/0x5c6 [<c014e58e>] vma_adjust+0x286/0x2d6 [<c014e762>] vma_merge+0xe1/0x165 [<c011d3d5>] finish_task_switch+0x30/0x66 [<c02cf368>] schedule+0x844/0x87a [<c011ac2d>] do_page_fault+0x0/0x5c6 [<c02d1bab>] error_code+0x2f/0x38 Code: 83 7b 10 00 74 0b 89 ca 89 d8 e8 fb fe ff ff
Quoted 7 lines
01
Quoted 1 line
c6 89 d8 e8 ac df fe ff 5b 89 f0 5e c3 56 53 89 c3
8b
Quoted 1 line
00 8b 72 44 f6 c4 08 74 08 <0f> 0b b2 01 ce 52 2e
c0
Quoted 1 line
85 f6 75 08 0f 0b b3 01 ce 52 2e c0 8b
____________________________________________________________________________________
8:00? 8:25? 8:40? Find a flick in no time with the Yahoo! Search movie showtime shortcut.
http://tools.search.yahoo.com/shortcuts/#news
Quoted 8 lines · Pat
Yes, there is a specific app which seems to be related to the kernel panic.
We use a few third party drivers on the system, so my initial suspicions were on: 3w_9xxx(U) which is the RAID card driver and fusedriver(U) which is a hardware PCI card driver.
Quoted 5 lines
The app which is kernel panicing is the only application which makes use of the fusedriver. I'd like to confirm my suspicions on the driver. Do you have any suggestions on how I could trace the kernel panic to that specific driver?
Indeed, yes, get its source and look to see what it's doing with PageReserved - or mail it to me privately and I'll take a look. But my Google for fusedriver didn't show anything, and clearly it has nothing to do with the FUSE filesystems.
Hugh
Quoted 17 lines · --- Hugh Dickins
Quoted 3 lines
The app which is kernel panicing is the only application which makes use of the fusedriver. I'd like to confirm my suspicions on the driver. Do
you
Quoted 1 line
have any suggestions on how I could trace the
kernel
Quoted 1 line
panic to that specific driver?
Indeed, yes, get its source and look to see what it's doing
with PageReserved - or mail it to me privately and I'll take
a look. But my Google for fusedriver didn't show anything,
and clearly it has nothing to do with the FUSE filesystems.
Quoted 2 lines
Hugh
Sorry for the late reply, I was on vacation with the family for a few days.
fusedriver is a driver for a customized fpga pci card used in our systems. I'm not sure how open the manufacturer will be in sending me the source for the drivers. I'll contact them though to get their input, specifically on their use of PageReserved.
Thanks,
Pat
____________________________________________________________________________________
Sucker-punch spam with award-winning protection. Try the free Yahoo! Mail Beta.
http://advision.webevents.yahoo.com/mailbeta/features_spam.html