2.6.10-rc2-bk13/16: Oops on Dual Opteron 246: Unable to handle kernel NULL pointer dereference
G'day
Happens at attempt to defragment InnoDB datafile by xfs_fsr.
lights innodb # xfs_fsr ib_logfile1 (of ~5G size) (all & only innodb-related files lie on the xfs partition)
Hardware Environment: Dual Opteron 246 / Iwill DK8S2 [AMD-8111] / 4G RAM/ LSI 320-2X
Software Environment:[ 2.6.10-rc2-bk13/16 reiserfs(mount as root),xfs ], gcc (GCC) 3.4.3 (Gentoo Linux 3.4.3, ssp-3.4.3-0, pie-8.7.6.6), glibc-2.3.4, xfs_fsr, mysql(with InnoDB)
Btw, I couldn't find any xfs staff in trace, and it seems reiserfs related more here. Oops 100% repeatable for us with 2.6.10-rc2-bk13 and 2.6.10-rc2-bk16
I send it to bugzilla.kernel.org:
http://bugme.osdl.org/show_bug.cgi?id=3854
Here is a trace (ksymoopsed)
lights root # Unable to handle kernel NULL pointer dereference at 0000000000000019 RIP:
<ffffffff801ca91c>{write_ordered_buffers+76}
PML4 e7bf3067 PGD e70bd067 PMD 0
Oops: 0002 [1] PREEMPT SMP
CPU 1
Pid: 160, comm: pdflush Not tainted 2.6.10-rc2-bk16
RIP: 0010:[<ffffffff801ca91c>] <ffffffff801ca91c>{write_ordered_buffers+76}
Using defaults from ksymoops -t elf64-x86-64 -a i386:x86-64
RSP: 0000:000001007fe35b58 EFLAGS: 00010293
RAX: 0000000000000000 RBX: 0000000000000001 RCX: 000001003b766910
RDX: 000000000000000c RSI: 000001007c8aa5b0 RDI: 000001007fe35b68
RBP: 000001005cb08ca8 R08: 0000000000000000 R09: 0000010002b41f28
R10: 0000010046497540 R11: ffffffff802965d0 R12: ffffff00000a1168
R13: 0000000000000000 R14: 000001007fe35b58 R15: 0000000000022f98
FS: 0000002a958624a0(0000) GS:ffffffff80464800(0000) knlGS:0000000000000000
CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 0000000000000019 CR3: 00000000f6f82000 CR4: 00000000000006e0
Stack: 000001003c3ce570 000001003df69f30 000001007c8aa190 000001007c8aa1e8
000001007c8aa240 000001007c8aa298 000001007c8aa2f0 000001007c8aa348
000001007c8aa3a0 000001007c8aa3f8
Call Trace:<ffffffff801cb0e5>{flush_commit_list+421}
<ffffffff801cf109>{do_journal_end+3129}
<ffffffff80325d0e>{schedule_timeout+174}
<ffffffff80147a70>{keventd_create_kthread+0}
<ffffffff801bc540>{reiserfs_write_super+64}
<ffffffff80179d70>{sync_supers+128}
<ffffffff8015727a>{wb_kupdate+42} <ffffffff80157f53>{pdflush+323}
<ffffffff80157250>{wb_kupdate+0} <ffffffff80157e10>{pdflush+0}
<ffffffff80147a29>{kthread+217} <ffffffff8010e057>{child_rip+8}
<ffffffff80147a70>{keventd_create_kthread+0}
<ffffffff80147950>{kthread+0}
<ffffffff8010e04f>{child_rip+0}
Code: f0 ff 43 18 f0 0f ba 2b 02 19 c0 85 c0 0f 84 91 00 00 00 8b Quoted 3 lines
Quoted 3 lines
RIP;
ffffffff801ca91c<write_ordered_buffers+4c/250> <=====R11;
ffffffff802965d0<as_merged_request+0/1d0>
Trace; ffffffff801cb0e5 <flush_commit_list+1a5/580> Trace; ffffffff80325d0e <schedule_timeout+ae/d0> Trace; ffffffff801bc540 <reiserfs_write_super+40/80> Trace; ffffffff8015727a <wb_kupdate+2a/130> Trace; ffffffff80157250 <wb_kupdate+0/130> Trace; ffffffff80147a29 <kthread+d9/120> Trace; ffffffff80147a70 <keventd_create_kthread+0/60> Trace; ffffffff8010e04f <child_rip+0/11>
Code; ffffffff801ca91c <write_ordered_buffers+4c/250> 0000000000000000 <_RIP>:
Code; ffffffff801ca91c <write_ordered_buffers+4c/250> <=====
0: f0 ff 43 18 lock incl 0x18(%rbx) <=====
Code; ffffffff801ca920 <write_ordered_buffers+50/250>
4: f0 0f ba 2b 02 lock btsl $0x2,(%rbx)
Code; ffffffff801ca925 <write_ordered_buffers+55/250>
9: 19 c0 sbb %eax,%eax
Code; ffffffff801ca927 <write_ordered_buffers+57/250>
b: 85 c0 test %eax,%eax
Code; ffffffff801ca929 <write_ordered_buffers+59/250>
d: 0f 84 91 00 00 00 je a4 <_RIP+0xa4>
Code; ffffffff801ca92f <write_ordered_buffers+5f/250>
13: 8b 00 mov (%rax),%eax
CR2: 0000000000000019
________________________________________________
.config => http://sysadminday.org.ru/3854/config
Signature · 3 lines
Best regards.
Alexander Y. Fomichev <[email protected]>
Public PGP key: http://sysadminday.org.ru/gluk.ascQuoted 88 lines · Alexander Y. Fomichev
G'day
Happens at attempt to defragment InnoDB datafile by
xfs_fsr.lights innodb #
xfs_fsrib_logfile1(of ~5G size) (all & only innodb-related files lie on the xfs partition)Hardware Environment: Dual Opteron 246 / Iwill DK8S2 [AMD-8111] / 4G RAM/ LSI 320-2X
Software Environment:[ 2.6.10-rc2-bk13/16 reiserfs(mount as root),xfs ], gcc (GCC) 3.4.3 (Gentoo Linux 3.4.3, ssp-3.4.3-0, pie-8.7.6.6), glibc-2.3.4,
xfs_fsr, mysql(with InnoDB)Btw, I couldn't find any xfs staff in trace, and it seems reiserfs related more here. Oops 100% repeatable for us with 2.6.10-rc2-bk13 and 2.6.10-rc2-bk16
I send it to bugzilla.kernel.org:
http://bugme.osdl.org/show_bug.cgi?id=3854Here is a trace (ksymoopsed)
lights root # Unable to handle kernel NULL pointer dereference at 0000000000000019 RIP:
<ffffffff801ca91c>{write_ordered_buffers+76}
PML4e7bf3067PGDe70bd067PMD 0Oops: 0002 [1] PREEMPT SMP CPU 1 Pid: 160, comm: pdflush Not tainted 2.6.10-rc2-bk16 RIP: 0010:[<ffffffff801ca91c>] <ffffffff801ca91c>{write_ordered_buffers+76} Using defaults from ksymoops -t elf64-x86-64 -a i386:x86-64 RSP: 0000:000001007fe35b58 EFLAGS: 00010293 RAX: 0000000000000000 RBX: 0000000000000001 RCX: 000001003b766910 RDX: 000000000000000c RSI: 000001007c8aa5b0 RDI: 000001007fe35b68 RBP: 000001005cb08ca8 R08: 0000000000000000 R09: 0000010002b41f28 R10: 0000010046497540 R11: ffffffff802965d0 R12: ffffff00000a1168 R13: 0000000000000000 R14: 000001007fe35b58 R15: 0000000000022f98 FS: 0000002a958624a0(0000) GS:ffffffff80464800(0000) knlGS:0000000000000000 CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b CR2: 0000000000000019 CR3: 00000000f6f82000 CR4: 00000000000006e0 Stack: 000001003c3ce570 000001003df69f30 000001007c8aa190 000001007c8aa1e8 000001007c8aa240 000001007c8aa298 000001007c8aa2f0 000001007c8aa348 000001007c8aa3a0 000001007c8aa3f8 Call Trace:<ffffffff801cb0e5>{flush_commit_list+421} <ffffffff801cf109>{do_journal_end+3129} <ffffffff80325d0e>{schedule_timeout+174} <ffffffff80147a70>{keventd_create_kthread+0} <ffffffff801bc540>{reiserfs_write_super+64} <ffffffff80179d70>{sync_supers+128} <ffffffff8015727a>{wb_kupdate+42} <ffffffff80157f53>{pdflush+323} <ffffffff80157250>{wb_kupdate+0} <ffffffff80157e10>{pdflush+0} <ffffffff80147a29>{kthread+217} <ffffffff8010e057>{child_rip+8} <ffffffff80147a70>{keventd_create_kthread+0} <ffffffff80147950>{kthread+0} <ffffffff8010e04f>{child_rip+0} Code: f0 ff 43 18 f0 0f ba 2b 02 19 c0 85 c0 0f 84 91 00 00 00 8bQuoted 3 lines
Quoted 3 lines
RIP;
ffffffff801ca91c<write_ordered_buffers+4c/250> <=====R11;
ffffffff802965d0<as_merged_request+0/1d0>Trace;
ffffffff801cb0e5<flush_commit_list+1a5/580> Trace;ffffffff80325d0e<schedule_timeout+ae/d0> Trace;ffffffff801bc540<reiserfs_write_super+40/80> Trace;ffffffff8015727a<wb_kupdate+2a/130> Trace;ffffffff80157250<wb_kupdate+0/130> Trace;ffffffff80147a29<kthread+d9/120> Trace;ffffffff80147a70<keventd_create_kthread+0/60> Trace;ffffffff8010e04f<child_rip+0/11>Code;
ffffffff801ca91c<write_ordered_buffers+4c/250> 0000000000000000 <_RIP>:
Code;ffffffff801ca91c<write_ordered_buffers+4c/250> <=====
0: f0 ff 43 18 lock incl 0x18(%rbx) <=====
Code;ffffffff801ca920<write_ordered_buffers+50/250>
4: f0 0f ba 2b 02 lock btsl $0x2,(%rbx)
Code;ffffffff801ca925<write_ordered_buffers+55/250>
9: 19 c0 sbb %eax,%eax
Code;ffffffff801ca927<write_ordered_buffers+57/250>
b: 85 c0 test %eax,%eax
Code;ffffffff801ca929<write_ordered_buffers+59/250>
d: 0f 84 91 00 00 00 je a4 <_RIP+0xa4>
Code;ffffffff801ca92f<write_ordered_buffers+5f/250>
13: 8b 00 mov (%rax),%eax
Quoted 5 lines
CR2: 0000000000000019
________________________________________________
.config => http://sysadminday.org.ru/3854/config
And here is a BUG() from the same machine
----------- [cut here ] --------- [please bite here ] ---------
Kernel BUG at journal:723
invalid operand: 0000 [1] PREEMPT SMP
CPU 0
Modules linked in: ipt_state ipt_REJECT iptable_filter
Pid: 162, comm: kswapd0 Not tainted 2.6.10-rc2-bk16
RIP: 0010:[<ffffffff801ca6b7>] <ffffffff801ca6b7>{reiserfs_free_jh+71}
RSP: 0018:00000100f6e51c58 EFLAGS: 00010246
RAX: 0000000000000000 RBX: 00000100f5749f50 RCX: 0000010080000000
RDX: 000000000033852d RSI: 0000000000000000 RDI: 000001007ff931c0
RBP: 00000100f5749f50 R08: 0000000000000000 R09: 0000000000000040
R10: 0000000000000000 R11: ffffffff801b5e30 R12: 00000100802e3fe8
R13: ffffff00000a1168 R14: 00000100a6d06ec0 R15: 0000000000000000
FS: 0000002a95a68090(0000) GS:ffffffff80464780(0000) knlGS:0000000000000000
CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 0000002a955ae000 CR3: 0000000000101000 CR4: 00000000000006e0
Process kswapd0 (pid: 162, threadinfo 00000100f6e50000, task 00000100f6e4a130)
Stack: 00000100f5749f50 ffffffff801b5ecb 0000000000000000 00000100802e3fe8
00000100a6d06ec0 0000000000000000 ffffffffffffffff ffffffff8015c9e1
00000100a6d06ec0 00000100802e3fe8
Call Trace:<ffffffff801b5ecb>{reiserfs_releasepage+155}
<ffffffff8015c9e1>{invalidate_complete_page+49}
<ffffffff8015cd84>{invalidate_mapping_pages+132}
<ffffffff8018d152>{shrink_icache_memory+306}
<ffffffff8015d0f2>{shrink_slab+210}
<ffffffff8015e786>{balance_pgdat+582}
<ffffffff8015e9aa>{kswapd+282}
<ffffffff80148050>{autoremove_wake_function+0}
<ffffffff8012c9d0>{finish_task_switch+64}
<ffffffff80148050>{autoremove_wake_function+0}
<ffffffff8012ca2e>{schedule_tail+14} <ffffffff8010e057>{child_rip+8}
<ffffffff8015e890>{kswapd+0} <ffffffff8010e04f>{child_rip+0} Code: 0f 0b a8 0a 34 80 ff ff ff ff d3 02 f0 ff 0d 06 97 28 00 f0
Signature · 2 lines
Sergey S. Kostyliov <[email protected]> Jabber ID: [email protected]