two Oopses on 2.6.10
I've had two recent Oopses on two different 2.6.10 kernels on an SMP system.
The system was being used "normally" -- a web browser, a few xterms and
text-editors, and a VNC window running on it, in addition to some services (apache, ssh...). The only thing that sticks out is that this only happens when someone's physically in front of the box, using its kb/mouse/monitor. We have this system on a KVM, and we get the jumping mouse a few times a day, and sometimes X and/or the whole system locks up... at first I thought this might be a KVM problem, but with these kernel oops I'm not sure.
The attached oops reports were created by copying the oops info from /var/log/messages into oops.01 and oops.02, then running:
# ksymoops -K -o /lib/modules/2.6.10-1.9_FC2smp/ -m /boot/System.map-2.6.10-1.9_FC2smp oops.01 >ksymoops.01
# ksymoops -K -o /lib/modules/2.6.10-1.12_FC2smp/ -m /boot/System.map-2.6.10-1.12_FC2smp oops.02 >ksymoops.02
Please let me know if there's any other information I need to provide.
-Anthony DiSante
http://nodivisions.com/
ksymoops 2.4.5 on i686 2.6.10-1.12_FC2. Options used
-V (default)
-K (specified)
-l /proc/modules (default)
-o /lib/modules/2.6.10-1.9_FC2smp/ (specified)
-m /boot/System.map-2.6.10-1.9_FC2smp (specified)No modules in ksyms, skipping objects
No ksyms, skipping lsmod
Jan 25 15:30:11 hostfoo kernel: eip: c0124efd Jan 25 15:30:11 hostfoo kernel: kernel BUG at include/asm/spinlock.h:146! Jan 25 15:30:11 hostfoo kernel: invalid operand: 0000 [#1] Jan 25 15:30:11 hostfoo kernel: CPU: 0 Jan 25 15:30:11 hostfoo kernel: EIP: 0060:[<c02bdc7d>] Not tainted VLI Using defaults from ksymoops -t elf32-i386 -a i386 Jan 25 15:30:11 hostfoo kernel: EFLAGS: 00210006 (2.6.10-1.9_FC2smp) Jan 25 15:30:11 hostfoo kernel: eax: c0124efd ebx: 00200286 ecx: c02d4a00 edx: c02d4a00 Jan 25 15:30:11 hostfoo kernel: esi: c201fa00 edi: cc6bbfa0 ebp: f576be80 esp: cc6bbf18 Jan 25 15:30:11 hostfoo kernel: ds: 007b es: 007b ss: 0068 Jan 25 15:30:11 hostfoo kernel: Stack: cc6bbf38 c201fa60 c0124efd cc6bbf38 181712b7 c0124fe9 181712b7 c02bd6a3 Jan 25 15:30:11 hostfoo kernel: c2020028 c2020028 181712b7 00000001 dead4ead 4b87ad6e c0125706 da7c1a40 Jan 25 15:30:11 hostfoo kernel: c201fa60 f576be80 0000002e cc6bbfa0 00000000 0000002e c016231d 00000000 Jan 25 15:30:11 hostfoo kernel: [<c0124efd>] del_timer+0x22/0x63 Jan 25 15:30:11 hostfoo kernel: [<c0124fe9>] del_singleshot_timer_sync+0x8/0x21 Jan 25 15:30:11 hostfoo kernel: [<c02bd6a3>] schedule_timeout+0x9a/0xae Jan 25 15:30:11 hostfoo kernel: [<c0125706>] process_timeout+0x0/0x5 Jan 25 15:30:11 hostfoo kernel: [<c016231d>] do_poll+0x88/0xa6 Jan 25 15:30:11 hostfoo kernel: [<c0162456>] sys_poll+0x11b/0x1bd Jan 25 15:30:11 hostfoo kernel: [<c0161963>] __pollwait+0x0/0x95 Jan 25 15:30:11 hostfoo kernel: [<c0121b34>] sys_gettimeofday+0x25/0x55 Jan 25 15:30:11 hostfoo kernel: [<c0103ccb>] syscall_call+0x7/0xb Jan 25 15:30:11 hostfoo kernel: Code: 81 00 00 00 00 01 c3 f0 ff 00 c3 56 89 c6 53 9c 5b fa 81 78 04 ad 4e ad de 74 18 ff 74 24 08 68 00 4a 2d c0 e8 c6 04 e6 ff 59 58 <0f> 0b 92 00 e0 39 2d c0 f0 fe 0e 79 13 f7 c3 00 02 00 00 74 01
Quoted 10 lines
Quoted 10 lines
EIP;
c02bdc7d<_spin_lock_irqsave+20/45> <=====eax;
c0124efd<del_timer+22/63>
ebx; 00200286 Before first symbol
ecx;c02d4a00<__func__.2+9dd7/2f1fb>
edx;c02d4a00<__func__.2+9dd7/2f1fb>
esi;c201fa00<END_OF_CODE+1bcea00/????>
edi;cc6bbfa0<END_OF_CODE+c26afa0/????>
ebp;f576be80<END_OF_CODE+3531ae80/????>
esp;cc6bbf18<END_OF_CODE+c26af18/????>
Code; c02bdc52 <_write_unlock+1/8>
00000000 <_EIP>:
Code; c02bdc52 <_write_unlock+1/8>
0: 81 00 00 00 00 01 addl $0x1000000,(%eax)
Code; c02bdc58 <_write_unlock+7/8>
6: c3 ret
Code; c02bdc59 <_read_unlock+0/4>
7: f0 ff 00 lock incl (%eax)
Code; c02bdc5c <_read_unlock+3/4>
a: c3 ret
Code; c02bdc5d <_spin_lock_irqsave+0/45>
b: 56 push %esi
Code; c02bdc5e <_spin_lock_irqsave+1/45>
c: 89 c6 mov %eax,%esi
Code; c02bdc60 <_spin_lock_irqsave+3/45>
e: 53 push %ebx
Code; c02bdc61 <_spin_lock_irqsave+4/45>
f: 9c pushf
Code; c02bdc62 <_spin_lock_irqsave+5/45>
10: 5b pop %ebx
Code; c02bdc63 <_spin_lock_irqsave+6/45>
11: fa cli
Code; c02bdc64 <_spin_lock_irqsave+7/45>
12: 81 78 04 ad 4e ad de cmpl $0xdead4ead,0x4(%eax)
Code; c02bdc6b <_spin_lock_irqsave+e/45>
19: 74 18 je 33 <_EIP+0x33>
Code; c02bdc6d <_spin_lock_irqsave+10/45>
1b: ff 74 24 08 pushl 0x8(%esp)
Code; c02bdc71 <_spin_lock_irqsave+14/45>
1f: 68 00 4a 2d c0 push $0xc02d4a00
Code; c02bdc76 <_spin_lock_irqsave+19/45>
24: e8 c6 04 e6 ff call ffe604ef <_EIP+0xffe604ef>
Code; c02bdc7b <_spin_lock_irqsave+1e/45>
29: 59 pop %ecx
Code; c02bdc7c <_spin_lock_irqsave+1f/45>
2a: 58 pop %eax
Code; c02bdc7d <_spin_lock_irqsave+20/45> <=====
2b: 0f 0b ud2a <=====
Code; c02bdc7f <_spin_lock_irqsave+22/45>
2d: 92 xchg %eax,%edx
Code; c02bdc80 <_spin_lock_irqsave+23/45>
2e: 00 e0 add %ah,%al
Code; c02bdc82 <_spin_lock_irqsave+25/45>
30: 39 2d c0 f0 fe 0e cmp %ebp,0xefef0c0
Code; c02bdc88 <_spin_lock_irqsave+2b/45>
36: 79 13 jns 4b <_EIP+0x4b>
Code; c02bdc8a <_spin_lock_irqsave+2d/45>
38: f7 c3 00 02 00 00 test $0x200,%ebx
Code; c02bdc90 <_spin_lock_irqsave+33/45>
3e: 74 01 je 41 <_EIP+0x41>
Jan 25 15:30:12 hostfoo kernel: <1>Unable to handle kernel paging request at virtual address 851ba4f4 Jan 25 15:30:12 hostfoo kernel: c01187f5 Jan 25 15:30:12 hostfoo kernel: *pde = 349a9001 Jan 25 15:30:12 hostfoo kernel: Oops: 0000 [#2] Jan 25 15:30:12 hostfoo kernel: CPU: 1 Jan 25 15:30:12 hostfoo kernel: EIP: 0060:[<c01187f5>] Not tainted VLI Jan 25 15:30:12 hostfoo kernel: EFLAGS: 00013082 (2.6.10-1.9_FC2smp) Jan 25 15:30:12 hostfoo kernel: eax: 31383135 ebx: c03a9100 ecx: 00000000 edx: da7c1a40 Jan 25 15:30:12 hostfoo kernel: esi: c03a9100 edi: f4f3dd38 ebp: f4f3dd00 esp: f4f3dcf0 Jan 25 15:30:12 hostfoo kernel: ds: 007b es: 007b ss: 0068 Jan 25 15:30:12 hostfoo kernel: Stack: da7c1a40 e5e6a044 00000001 da7c1a40 f4f3dd48 c0118d16 f4f3dd40 c011a3f7 Jan 25 15:30:12 hostfoo kernel: 00000000 00000001 00000380 c2027060 00000001 00000000 00000001 00000001 Jan 25 15:30:12 hostfoo kernel: 00000000 00000001 00003082 e5e6a044 00000001 e0e7c518 f4f3dd6c c011a44f Jan 25 15:30:12 hostfoo kernel: [<c0118d16>] try_to_wake_up+0x20/0x21e Jan 25 15:30:12 hostfoo kernel: [<c011a3f7>] scheduler_tick+0x3b3/0x3c9 Jan 25 15:30:12 hostfoo kernel: [<c011a44f>] __wake_up_common+0x36/0x5b Jan 25 15:30:12 hostfoo kernel: [<c011a49d>] __wake_up+0x29/0x3c Jan 25 15:30:12 hostfoo kernel: [<c0268de4>] sock_def_readable+0x31/0x5e Jan 25 15:30:12 hostfoo kernel: [<c02b9353>] unix_stream_sendmsg+0x253/0x348 Jan 25 15:30:12 hostfoo kernel: [<c02662b6>] sock_sendmsg+0xdb/0xf7 Jan 25 15:30:12 hostfoo kernel: [<c01199a1>] find_busiest_group+0xf1/0x2d2 Jan 25 15:30:12 hostfoo kernel: [<c019ca7e>] avc_has_perm_noaudit+0x37/0xdc Jan 25 15:30:12 hostfoo kernel: [<c012e826>] autoremove_wake_function+0x0/0x2d Jan 25 15:30:12 hostfoo kernel: [<c019dbf4>] inode_has_perm+0x4c/0x54 Jan 25 15:30:12 hostfoo kernel: [<c026672f>] sock_readv_writev+0x7c/0x83 Jan 25 15:30:12 hostfoo kernel: [<c0266794>] sock_writev+0x2a/0x31 Jan 25 15:30:12 hostfoo kernel: [<c026676a>] sock_writev+0x0/0x31 Jan 25 15:30:12 hostfoo kernel: [<c0152702>] do_readv_writev+0x15d/0x1de Jan 25 15:30:12 hostfoo kernel: [<c0109af4>] convert_fxsr_from_user+0x27/0xe9 Jan 25 15:30:12 hostfoo kernel: [<c011a3f7>] scheduler_tick+0x3b3/0x3c9 Jan 25 15:30:12 hostfoo kernel: [<c0109d9c>] restore_i387+0x17/0x7a Jan 25 15:30:12 hostfoo kernel: [<c01527fe>] vfs_writev+0x3d/0x41 Jan 25 15:30:12 hostfoo kernel: [<c01528a0>] sys_writev+0x3c/0x62 Jan 25 15:30:12 hostfoo kernel: [<c0103ccb>] syscall_call+0x7/0xb Jan 25 15:30:12 hostfoo kernel: Code: ba 05 00 00 00 39 c2 0f 4d c2 c3 55 89 e5 57 89 d7 56 53 51 89 45 f0 9c 8f 07 fa 8b 55 f0 bb 00 91 3a c0 89 de 8b 42 04 8b 40 10 <03> 34 85 20 e0 3a c0 89 f0 e8 b6 53 1a 00 8b 55 f0 8b 42 04 8b
Quoted 9 lines
Quoted 9 lines
EIP;
c01187f5<task_rq_lock+20/55> <=====eax; 31383135 Before first symbol
ebx;c03a9100<per_cpu__runqueues+0/960>
edx;da7c1a40<END_OF_CODE+1a370a40/????>
esi;c03a9100<per_cpu__runqueues+0/960>
edi;f4f3dd38<END_OF_CODE+34aecd38/????>
ebp;f4f3dd00<END_OF_CODE+34aecd00/????>
esp;f4f3dcf0<END_OF_CODE+34aeccf0/????>
Code; c01187ca <task_timeslice+22/2d>
00000000 <_EIP>:
Code; c01187ca <task_timeslice+22/2d>
0: ba 05 00 00 00 mov $0x5,%edx
Code; c01187cf <task_timeslice+27/2d>
5: 39 c2 cmp %eax,%edx
Code; c01187d1 <task_timeslice+29/2d>
7: 0f 4d c2 cmovge %edx,%eax
Code; c01187d4 <task_timeslice+2c/2d>
a: c3 ret
Code; c01187d5 <task_rq_lock+0/55>
b: 55 push %ebp
Code; c01187d6 <task_rq_lock+1/55>
c: 89 e5 mov %esp,%ebp
Code; c01187d8 <task_rq_lock+3/55>
e: 57 push %edi
Code; c01187d9 <task_rq_lock+4/55>
f: 89 d7 mov %edx,%edi
Code; c01187db <task_rq_lock+6/55>
11: 56 push %esi
Code; c01187dc <task_rq_lock+7/55>
12: 53 push %ebx
Code; c01187dd <task_rq_lock+8/55>
13: 51 push %ecx
Code; c01187de <task_rq_lock+9/55>
14: 89 45 f0 mov %eax,0xfffffff0(%ebp)
Code; c01187e1 <task_rq_lock+c/55>
17: 9c pushf
Code; c01187e2 <task_rq_lock+d/55>
18: 8f 07 popl (%edi)
Code; c01187e4 <task_rq_lock+f/55>
1a: fa cli
Code; c01187e5 <task_rq_lock+10/55>
1b: 8b 55 f0 mov 0xfffffff0(%ebp),%edx
Code; c01187e8 <task_rq_lock+13/55>
1e: bb 00 91 3a c0 mov $0xc03a9100,%ebx
Code; c01187ed <task_rq_lock+18/55>
23: 89 de mov %ebx,%esi
Code; c01187ef <task_rq_lock+1a/55>
25: 8b 42 04 mov 0x4(%edx),%eax
Code; c01187f2 <task_rq_lock+1d/55>
28: 8b 40 10 mov 0x10(%eax),%eax
Code; c01187f5 <task_rq_lock+20/55> <=====
2b: 03 34 85 20 e0 3a c0 add 0xc03ae020(,%eax,4),%esi <=====
Code; c01187fc <task_rq_lock+27/55>
32: 89 f0 mov %esi,%eax
Code; c01187fe <task_rq_lock+29/55>
34: e8 b6 53 1a 00 call 1a53ef <_EIP+0x1a53ef>
Code; c0118803 <task_rq_lock+2e/55>
39: 8b 55 f0 mov 0xfffffff0(%ebp),%edx
Code; c0118806 <task_rq_lock+31/55>
3c: 8b 42 04 mov 0x4(%edx),%eax
Code; c0118809 <task_rq_lock+34/55>
3f: 8b .byte 0x8b
ksymoops 2.4.5 on i686 2.6.10-1.12_FC2. Options used
-V (default)
-K (specified)
-l /proc/modules (default)
-o /lib/modules/2.6.10-1.12_FC2smp/ (specified)
-m /boot/System.map-2.6.10-1.12_FC2smp (specified)No modules in ksyms, skipping objects
No ksyms, skipping lsmod
Feb 11 01:34:33 hostfoo kernel: <1>Unable to handle kernel paging request at virtual address 80899e10 Feb 11 01:34:33 hostfoo kernel: c02be356 Feb 11 01:34:33 hostfoo kernel: *pde = f6d6f424 Feb 11 01:34:33 hostfoo kernel: Oops: 0000 [#2] Feb 11 01:34:33 hostfoo kernel: CPU: 1 Feb 11 01:34:33 hostfoo kernel: EIP: 0060:[<c02be356>] Not tainted VLI Using defaults from ksymoops -t elf32-i386 -a i386 Feb 11 01:34:33 hostfoo kernel: EFLAGS: 00013087 (2.6.10-1.12_FC2smp) Feb 11 01:34:33 hostfoo kernel: eax: 80899e0c ebx: 80899e0c ecx: 00000000 edx: dedc7540 Feb 11 01:34:33 hostfoo kernel: esi: 80899e0c edi: e185ed38 ebp: e185ed00 esp: e185ece8 Feb 11 01:34:33 hostfoo kernel: ds: 007b es: 007b ss: 0068 Feb 11 01:34:33 hostfoo kernel: Stack: c03aa100 c01188ab dedc7540 e2ac1044 00000001 dedc7540 e185ed48 c0118dbe Feb 11 01:34:33 hostfoo kernel: 00000037 00000089 00000000 00000001 00000200 c2027060 00000001 00000000 Feb 11 01:34:33 hostfoo kernel: 00000001 00000001 00000000 00000001 00003082 e2ac1044 00000001 de88f518 Feb 11 01:34:33 hostfoo kernel: [<c01188ab>] task_rq_lock+0x2e/0x55 Feb 11 01:34:33 hostfoo kernel: [<c0118dbe>] try_to_wake_up+0x20/0x21e Feb 11 01:34:33 hostfoo kernel: [<c011a4f7>] __wake_up_common+0x36/0x5b Feb 11 01:34:33 hostfoo kernel: [<c011a545>] __wake_up+0x29/0x3c Feb 11 01:34:33 hostfoo kernel: [<c026970c>] sock_def_readable+0x31/0x5e Feb 11 01:34:33 hostfoo kernel: [<c02b9aaf>] unix_stream_sendmsg+0x253/0x348 Feb 11 01:34:33 hostfoo kernel: [<c011a545>] __wake_up+0x29/0x3c Feb 11 01:34:33 hostfoo kernel: [<c0266b12>] sock_sendmsg+0xdb/0xf7 Feb 11 01:34:33 hostfoo kernel: [<c0119a49>] find_busiest_group+0xf1/0x2d2 Feb 11 01:34:33 hostfoo kernel: [<c019d4a2>] avc_has_perm_noaudit+0x37/0xdc Feb 11 01:34:33 hostfoo kernel: [<c012ec8e>] autoremove_wake_function+0x0/0x2d Feb 11 01:34:33 hostfoo kernel: [<c019e618>] inode_has_perm+0x4c/0x54 Feb 11 01:34:33 hostfoo kernel: [<c0266f8b>] sock_readv_writev+0x7c/0x83 Feb 11 01:34:33 hostfoo kernel: [<c0266ff0>] sock_writev+0x2a/0x31 Feb 11 01:34:33 hostfoo kernel: [<c0266fc6>] sock_writev+0x0/0x31 Feb 11 01:34:33 hostfoo kernel: [<c015302a>] do_readv_writev+0x15d/0x1de Feb 11 01:34:33 hostfoo kernel: [<c0109b1e>] convert_fxsr_from_user+0x27/0xe9 Feb 11 01:34:33 hostfoo kernel: [<c0109dc6>] restore_i387+0x17/0x7a Feb 11 01:34:33 hostfoo kernel: [<c0153126>] vfs_writev+0x3d/0x41 Feb 11 01:34:33 hostfoo kernel: [<c01531c8>] sys_writev+0x3c/0x62 Feb 11 01:34:33 hostfoo kernel: [<c0103ccb>] syscall_call+0x7/0xb Feb 11 01:34:33 hostfoo kernel: Code: c0 84 d2 0f 9f c0 c3 89 c2 f0 81 28 00 00 00 01 0f 94 c0 84 c0 b9 01 00 00 00 75 09 f0 81 02 00 00 00 01 31 c9 89 c8 c3 53 89 c3 <81> 78 04 ad 4e ad de 74 18 ff 74 24 04 68 80 4d 2d c0 e8 cf fe
Quoted 9 lines
Quoted 9 lines
EIP;
c02be356<_spin_lock+3/34> <=====eax;
80899e0cBefore first symbol
ebx;80899e0cBefore first symbol
edx;dedc7540<END_OF_CODE+1e974540/????>
esi;80899e0cBefore first symbol
edi;e185ed38<END_OF_CODE+2140bd38/????>
ebp;e185ed00<END_OF_CODE+2140bd00/????>
esp;e185ece8<END_OF_CODE+2140bce8/????>
Code; c02be32b <_spin_trylock+5/c>
00000000 <_EIP>:
Code; c02be32b <_spin_trylock+5/c>
0: c0 84 d2 0f 9f c0 c3 rolb $0x89,0xc3c09f0f(%edx,%edx,8)
Code; c02be332 <_write_trylock+0/21>
7: 89
Code; c02be333 <_write_trylock+1/21>
8: c2 f0 81 ret $0x81f0
Code; c02be336 <_write_trylock+4/21>
b: 28 00 sub %al,(%eax)
Code; c02be338 <_write_trylock+6/21>
d: 00 00 add %al,(%eax)
Code; c02be33a <_write_trylock+8/21>
f: 01 0f add %ecx,(%edi)
Code; c02be33c <_write_trylock+a/21>
11: 94 xchg %eax,%esp
Code; c02be33d <_write_trylock+b/21>
12: c0 84 c0 b9 01 00 00 rolb $0x0,0x1b9(%eax,%eax,8)
Code; c02be344 <_write_trylock+12/21>
19: 00
Code; c02be345 <_write_trylock+13/21>
1a: 75 09 jne 25 <_EIP+0x25>
Code; c02be347 <_write_trylock+15/21>
1c: f0 81 02 00 00 00 01 lock addl $0x1000000,(%edx)
Code; c02be34e <_write_trylock+1c/21>
23: 31 c9 xor %ecx,%ecx
Code; c02be350 <_write_trylock+1e/21>
25: 89 c8 mov %ecx,%eax
Code; c02be352 <_write_trylock+20/21>
27: c3 ret
Code; c02be353 <_spin_lock+0/34>
28: 53 push %ebx
Code; c02be354 <_spin_lock+1/34>
29: 89 c3 mov %eax,%ebx
Code; c02be356 <_spin_lock+3/34> <=====
2b: 81 78 04 ad 4e ad de cmpl $0xdead4ead,0x4(%eax) <=====
Code; c02be35d <_spin_lock+a/34>
32: 74 18 je 4c <_EIP+0x4c>
Code; c02be35f <_spin_lock+c/34>
34: ff 74 24 04 pushl 0x4(%esp)
Code; c02be363 <_spin_lock+10/34>
38: 68 80 4d 2d c0 push $0xc02d4d80
Code; c02be368 <_spin_lock+15/34>
3d: e8 .byte 0xe8
Code; c02be369 <_spin_lock+16/34>
3e: cf iret
Code; c02be36a <_spin_lock+17/34>
3f: fe .byte 0xfe
Quoted 2 lines · Anthony DiSante
# ksymoops -K -o /lib/modules/2.6.10-1.9_FC2smp/ -m /boot/System.map-2.6.10-1.9_FC2smp oops.01 >ksymoops.01
You seem to be using a Redhat/Fedora kernel and not a stock 2.6.10 - Two options - Either report this bug at https://bugzilla.redhat.com or reproduce it with a recent stock kernel like 2.6.11-rc4 and report it if you do reproduce it.
Parag