ext3 oops (probably I/O congestion/starvation related), already solved?
Hi,
I should like to ask if following oops has already been fixed in more recent kernels?
I'm seing this on a productive machine with a SuSE 9.1 kernel (2.6.5-7.108-default). It's tainted with the nvidia module but the oops is not related to it.
If this has been fixed in later kernels, please tell me so, I can then justify a downtime. I'm going through the archives and bk-patches myself but I've a backlog of about 2 months.
Unable to handle kernel paging request at virtual address 00200200 c0126c38 *pde = 00000000 Oops: 0002 [#1] CPU: 0 EIP: 0060:[<c0126c38>] Tainted: PF U
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00210082 (2.6.5-7.108-default)
eax: f68a41c8 ebx: e8e7bfc0 ecx: f6910980 edx: 00200200
esi: 00200246 edi: 004a04b0 ebp: 00000001 esp: d9d55d24
ds: 007b es: 007b ss: 0068
Stack: 00001388 0160076d f6910980 f5df8c80 f8822ca9 00000000 f6910980 0000005c
eee2d424 0000046a 00000001 00000000 00200086 d8a23d8c 00000000
f5df8cdc
d9d55d90 00000000 c19bcc10 c011f1c0 d9d55d94 d9d55d94 00200246
f3259200
Call Trace:
[<f8822ca9>] start_this_handle+0x309/0x480 [jbd]
[<c011f1c0>] autoremove_wake_function+0x0/0x30
[<c011f1c0>] autoremove_wake_function+0x0/0x30
[<f8826d72>] __log_start_commit+0x62/0x70 [jbd]
[<f8822ead>] journal_restart+0x8d/0x120 [jbd]
[<f8857c80>] ext3_clear_blocks+0x60/0x130 [ext3]
[<f8857e57>] ext3_free_data+0x107/0x140 [ext3]
[<f8857f5d>] ext3_free_branches+0xcd/0x1f0 [ext3]
[<f8857f5d>] ext3_free_branches+0xcd/0x1f0 [ext3]
[<f8858944>] ext3_truncate+0x8c4/0x9b0 [ext3]
[<f88271ee>] log_wait_commit+0x10e/0x170 [jbd]
[<c011b34e>] __wake_up+0xe/0x20
[<f8821c0c>] journal_stop+0x1bc/0x2b0 [jbd]
[<f8858ad7>] ext3_delete_inode+0xa7/0xd9 [ext3]
[<f8858a30>] ext3_delete_inode+0x0/0xd9 [ext3]
[<c0170b3c>] generic_delete_inode+0x8c/0x120
[<c016f93d>] iput+0x4d/0x70
[<c0168199>] sys_unlink+0x159/0x1a0
[<c0107dc9>] sysenter_past_esp+0x52/0x79
Code: 89 02 c7 43 04 00 02 20 00 c7 03 00 01 10 00 89 7b 08 89 daQuoted 9 lines
Quoted 9 lines
EIP;
c0126c38<__mod_timer+38/70> <=====eax;
f68a41c8<__crc_acpi_get_possible_resources+a56e/1ebb8> ebx;e8e7bfc0<__crc_inet_addr_type+1884d9/2c3f96> ecx;f6910980<__crc_ide_setup_pci_noise+25946/48c0e> edx; 00200200 <__crc___kill_fasync+13efec/1abbaf> esi; 00200246 <__crc___kill_fasync+13f032/1abbaf> edi;004a04b0<__crc_rtnetlink_links+4e7b2/8e103> esp;d9d55d24<__crc_unregister_binfmt+17f4b9/2e10a7>
Trace; f8822ca9 <__crc_xfrm_state_get_afinfo+6bb89/b1bca> Trace; c011f1c0 <autoremove_wake_function+0/30> Trace; c011f1c0 <autoremove_wake_function+0/30> Trace; f8826d72 <__crc_xfrm_state_get_afinfo+6fc52/b1bca> Trace; f8822ead <__crc_xfrm_state_get_afinfo+6bd8d/b1bca> Trace; f8857c80 <__crc_xfrm_state_get_afinfo+a0b60/b1bca> Trace; f8857e57 <__crc_xfrm_state_get_afinfo+a0d37/b1bca> Trace; f8857f5d <__crc_xfrm_state_get_afinfo+a0e3d/b1bca> Trace; f8857f5d <__crc_xfrm_state_get_afinfo+a0e3d/b1bca> Trace; f8858944 <__crc_xfrm_state_get_afinfo+a1824/b1bca> Trace; f88271ee <__crc_xfrm_state_get_afinfo+700ce/b1bca> Trace; c011b34e <__wake_up+e/20>
Trace; f8821c0c <__crc_xfrm_state_get_afinfo+6aaec/b1bca> Trace; f8858ad7 <__crc_xfrm_state_get_afinfo+a19b7/b1bca> Trace; f8858a30 <__crc_xfrm_state_get_afinfo+a1910/b1bca> Trace; c0170b3c <generic_delete_inode+8c/120> Trace; c016f93d <iput+4d/70>
Trace; c0168199 <sys_unlink+159/1a0>
Trace; c0107dc9 <sysenter_past_esp+52/79>
Code; c0126c38 <__mod_timer+38/70>
00000000 <_EIP>:
Code; c0126c38 <__mod_timer+38/70> <=====
0: 89 02 mov %eax,(%edx) <=====
Code; c0126c3a <__mod_timer+3a/70>
2: c7 43 04 00 02 20 00 movl $0x200200,0x4(%ebx)
Code; c0126c41 <__mod_timer+41/70>
9: c7 03 00 01 10 00 movl $0x100100,(%ebx)
Code; c0126c47 <__mod_timer+47/70>
f: 89 7b 08 mov %edi,0x8(%ebx)
Code; c0126c4a <__mod_timer+4a/70>
12: 89 da mov %ebx,%edx
kernel BUG at fs/ext3/super.c:412!
invalid operand: 0000 [#2]
CPU: 0
EIP: 0060:[<f885d19b>] Tainted: PF U
EFLAGS: 00010286 (2.6.5-7.108-default)
eax: 0000005e ebx: f5df8704 ecx: c033f764 edx: 00005cde
esi: f5df8680 edi: f5df8704 ebp: ed3d7400 esp: d8a23ee8
ds: 007b es: 007b ss: 0068
Stack: f8863680 f8862751 f8864ba6 0000019c f8864bdb ed3d7400 ed3d744c f886c000
d8a23f44 c015f246 ed3d7400 f6ec2b00 00000000 c015f26f ed3d7400
f886bfc0
c015ead8 f7ffea60 d8a23f54 c01737d0 ed3d7400 f7ff48c0 c014b33e
00000000
Call Trace:
[<c015f246>] generic_shutdown_super+0xe6/0x100
[<c015f26f>] kill_block_super+0xf/0x30
[<c015ead8>] deactivate_super+0x38/0x50
[<c01737d0>] sys_umount+0x70/0x2e0
[<c014b33e>] split_vma+0x18e/0x240
[<c0173a4c>] sys_oldumount+0xc/0xf
[<c0107dc9>] sysenter_past_esp+0x52/0x79
Code: 0f 0b 9c 01 a6 4b 86 f8 83 c4 14 eb 88 8b 46 2c 8b 88 e8 00Quoted 8 lines
Quoted 8 lines
EIP;
f885d19b<__crc_xfrm_state_get_afinfo+a607b/b1bca> <=====ebx;
f5df8704<__crc_change_page_attr+2c361f/43bc73> ecx;c033f764<console_sem+0/10>
esi;f5df8680<__crc_change_page_attr+2c359b/43bc73> edi;f5df8704<__crc_change_page_attr+2c361f/43bc73> ebp;ed3d7400<__crc_pm_send_all+7dd753/902541> esp;d8a23ee8<__crc_ckrm_class_set_shares+9a156/a0e03>
Trace; c015f246 <generic_shutdown_super+e6/100>
Trace; c015f26f <kill_block_super+f/30>
Trace; c015ead8 <deactivate_super+38/50>
Trace; c01737d0 <sys_umount+70/2e0>
Trace; c014b33e <split_vma+18e/240>
Trace; c0173a4c <sys_oldumount+c/f>
Trace; c0107dc9 <sysenter_past_esp+52/79>
Code; f885d19b <__crc_xfrm_state_get_afinfo+a607b/b1bca> 00000000 <_EIP>:
Code; f885d19b <__crc_xfrm_state_get_afinfo+a607b/b1bca> <=====
0: 0f 0b ud2a <=====
Code; f885d19d <__crc_xfrm_state_get_afinfo+a607d/b1bca>
2: 9c pushf
Code; f885d19e <__crc_xfrm_state_get_afinfo+a607e/b1bca>
3: 01 a6 4b 86 f8 83 add %esp,0x83f8864b(%esi)
Code; f885d1a4 <__crc_xfrm_state_get_afinfo+a6084/b1bca>
9: c4 14 eb les (%ebx,%ebp,8),%edx
Code; f885d1a7 <__crc_xfrm_state_get_afinfo+a6087/b1bca>
c: 88 8b 46 2c 8b 88 mov %cl,0x888b2c46(%ebx)
Code; f885d1ad <__crc_xfrm_state_get_afinfo+a608d/b1bca>
12: e8 00 00 00 00 call 17 <_EIP+0x17>
Unable to handle kernel paging request at virtual address 00200200 printing eip: c0126c38 *pde = 00000000 Oops: 0002 [#1] CPU: 0 EIP: 0060:[<c0126c38>] Tainted: PF U EFLAGS: 00210082 (2.6.5-7.108-default)
EIP is at __mod_timer+0x38/0x70
eax: f68a41c8 ebx: e8e7bfc0 ecx: f6910980 edx: 00200200
esi: 00200246 edi: 004a04b0 ebp: 00000001 esp: d9d55d24
ds: 007b es: 007b ss: 0068
Process rm (pid: 6103, threadinfo=d9d54000 task=c19bcc10)
Stack: 00001388 0160076d f6910980 f5df8c80 f8822ca9 00000000 f6910980 0000005c
eee2d424 0000046a 00000001 00000000 00200086 d8a23d8c 00000000
f5df8cdc
d9d55d90 00000000 c19bcc10 c011f1c0 d9d55d94 d9d55d94 00200246
f3259200
Call Trace:
[<f8822ca9>] start_this_handle+0x309/0x480 [jbd]
[<c011f1c0>] autoremove_wake_function+0x0/0x30
[<c011f1c0>] autoremove_wake_function+0x0/0x30
[<f8826d72>] __log_start_commit+0x62/0x70 [jbd]
[<f8822ead>] journal_restart+0x8d/0x120 [jbd]
[<f8857c80>] ext3_clear_blocks+0x60/0x130 [ext3]
[<f8857e57>] ext3_free_data+0x107/0x140 [ext3]
[<f8857f5d>] ext3_free_branches+0xcd/0x1f0 [ext3]
[<f8857f5d>] ext3_free_branches+0xcd/0x1f0 [ext3]
[<f8858944>] ext3_truncate+0x8c4/0x9b0 [ext3]
[<f88271ee>] log_wait_commit+0x10e/0x170 [jbd]
[<c011b34e>] __wake_up+0xe/0x20
[<f8821c0c>] journal_stop+0x1bc/0x2b0 [jbd]
[<f8858ad7>] ext3_delete_inode+0xa7/0xd9 [ext3]
[<f8858a30>] ext3_delete_inode+0x0/0xd9 [ext3]
[<c0170b3c>] generic_delete_inode+0x8c/0x120
[<c016f93d>] iput+0x4d/0x70
[<c0168199>] sys_unlink+0x159/0x1a0
[<c0107dc9>] sysenter_past_esp+0x52/0x79
Code: 89 02 c7 43 04 00 02 20 00 c7 03 00 01 10 00 89 7b 08 89 da
<3>sb orphan head is 16924729
sb_info orphan list:
inode sda1:16924729 at f5e7c914: mode 100600, nlink 0, next 0
Assertion failure in ext3_put_super() at fs/ext3/super.c:412: "list_empty(&sbi->s_orphan)"
------------[ cut here ]------------
kernel BUG at fs/ext3/super.c:412!
invalid operand: 0000 [#2]
CPU: 0
EIP: 0060:[<f885d19b>] Tainted: PF U
EFLAGS: 00010286 (2.6.5-7.108-default)EIP is at ext3_put_super+0xeb/0x1b0 [ext3]
eax: 0000005e ebx: f5df8704 ecx: c033f764 edx: 00005cde
esi: f5df8680 edi: f5df8704 ebp: ed3d7400 esp: d8a23ee8
ds: 007b es: 007b ss: 0068
Process submountd (pid: 6102, threadinfo=d8a22000 task=f66f8200)
Stack: f8863680 f8862751 f8864ba6 0000019c f8864bdb ed3d7400 ed3d744c f886c000
d8a23f44 c015f246 ed3d7400 f6ec2b00 00000000 c015f26f ed3d7400
f886bfc0
c015ead8 f7ffea60 d8a23f54 c01737d0 ed3d7400 f7ff48c0 c014b33e
00000000
Call Trace:
[<c015f246>] generic_shutdown_super+0xe6/0x100
[<c015f26f>] kill_block_super+0xf/0x30
[<c015ead8>] deactivate_super+0x38/0x50
[<c01737d0>] sys_umount+0x70/0x2e0
[<c014b33e>] split_vma+0x18e/0x240
[<c0173a4c>] sys_oldumount+0xc/0xf
[<c0107dc9>] sysenter_past_esp+0x52/0x79
Code: 0f 0b 9c 01 a6 4b 86 f8 83 c4 14 eb 88 8b 46 2c 8b 88 e8 00
<7>ieee1394: Node changed: 0-01:1023 -> 0-00:1023ieee1394: Node suspended: ID:BUS[0-00:1023] GUID[0010b9010140c170]
It's a firewire disk (160GB) mounted through a TSB43AB23 IEEE-1394a-2000 Controller. Mount options are:
/dev/sda1 on /media/ieee1394_sbp2-0010b9010140c170-0-0p1 type subfs (rw,noexec,nosuid,nodev,sync,procuid,iocharset=utf8)
It happened while trying to remove a 1GB file. Generally unlink() doesn't seem to be liked by this kernel.
Best regards,
Roberto Nibali, ratz
Signature · 1 line
echo '[q]sa[ln0=aln256%Pln256/snlbx]sb3135071790101768542287578439snlbxq' | dc
Quoted 8 lines · Roberto Nibali
Hi,
I should like to ask if following oops has already been fixed in more recent kernels?
I'm seing this on a productive machine with a SuSE 9.1 kernel (2.6.5-7.108-default). It's tainted with the nvidia module but the oops is not related to it.
Just to be on the safe side, have you reproduced it without? Just because nvidia doesn't show up in the trace, doesn't mean it hasn't corrupted memory elsewhere.
Quoted 34 lines
If this has been fixed in later kernels, please tell me so, I can then justify a downtime. I'm going through the archives and bk-patches myself but I've a backlog of about 2 months.
Unable to handle kernel paging request at virtual address 00200200 c0126c38 *pde = 00000000 Oops: 0002 [#1] CPU: 0 EIP: 0060:[<c0126c38>] Tainted: PF UUsing defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00210082 (2.6.5-7.108-default)
eax:f68a41c8ebx:e8e7bfc0ecx:f6910980edx: 00200200
esi: 00200246 edi:004a04b0ebp: 00000001 esp:d9d55d24
ds: 007b es: 007b ss: 0068Stack: 00001388 0160076d f6910980 f5df8c80 f8822ca9 00000000 f6910980 0000005c eee2d424 0000046a 00000001 00000000 00200086 d8a23d8c 00000000 f5df8cdc d9d55d90 00000000 c19bcc10 c011f1c0 d9d55d94 d9d55d94 00200246 f3259200 Call Trace: [<f8822ca9>] start_this_handle+0x309/0x480 [jbd] [<c011f1c0>] autoremove_wake_function+0x0/0x30 [<c011f1c0>] autoremove_wake_function+0x0/0x30 [<f8826d72>] __log_start_commit+0x62/0x70 [jbd] [<f8822ead>] journal_restart+0x8d/0x120 [jbd] [<f8857c80>] ext3_clear_blocks+0x60/0x130 [ext3] [<f8857e57>] ext3_free_data+0x107/0x140 [ext3] [<f8857f5d>] ext3_free_branches+0xcd/0x1f0 [ext3] [<f8857f5d>] ext3_free_branches+0xcd/0x1f0 [ext3] [<f8858944>] ext3_truncate+0x8c4/0x9b0 [ext3] [<f88271ee>] log_wait_commit+0x10e/0x170 [jbd] [<c011b34e>] __wake_up+0xe/0x20
Quoted 11 lines
[<f8821c0c>] journal_stop+0x1bc/0x2b0 [jbd] [<f8858ad7>] ext3_delete_inode+0xa7/0xd9 [ext3] [<f8858a30>] ext3_delete_inode+0x0/0xd9 [ext3] [<c0170b3c>] generic_delete_inode+0x8c/0x120 [<c016f93d>] iput+0x4d/0x70 [<c0168199>] sys_unlink+0x159/0x1a0 [<c0107dc9>] sysenter_past_esp+0x52/0x79 Code: 89 02 c7 43 04 00 02 20 00 c7 03 00 01 10 00 89 7b 08 89 daQuoted 1 line
Quoted 1 line
EIP;
c0126c38<__mod_timer+38/70> <=====
00200200 is the list deletion poison, double remove of a list and in this case the timer base. That's pretty bad news.
So please do try and reproduce without the nvidia module.
Quoted 41 lines
Quoted 7 lines
Quoted 7 lines
eax;
f68a41c8<__crc_acpi_get_possible_resources+a56e/1ebb8> ebx;e8e7bfc0<__crc_inet_addr_type+1884d9/2c3f96> ecx;f6910980<__crc_ide_setup_pci_noise+25946/48c0e> edx; 00200200 <__crc___kill_fasync+13efec/1abbaf> esi; 00200246 <__crc___kill_fasync+13f032/1abbaf> edi;004a04b0<__crc_rtnetlink_links+4e7b2/8e103> esp;d9d55d24<__crc_unregister_binfmt+17f4b9/2e10a7>Trace;
f8822ca9<__crc_xfrm_state_get_afinfo+6bb89/b1bca> Trace;c011f1c0<autoremove_wake_function+0/30> Trace;c011f1c0<autoremove_wake_function+0/30> Trace;f8826d72<__crc_xfrm_state_get_afinfo+6fc52/b1bca> Trace;f8822ead<__crc_xfrm_state_get_afinfo+6bd8d/b1bca> Trace;f8857c80<__crc_xfrm_state_get_afinfo+a0b60/b1bca> Trace;f8857e57<__crc_xfrm_state_get_afinfo+a0d37/b1bca> Trace;f8857f5d<__crc_xfrm_state_get_afinfo+a0e3d/b1bca> Trace;f8857f5d<__crc_xfrm_state_get_afinfo+a0e3d/b1bca> Trace;f8858944<__crc_xfrm_state_get_afinfo+a1824/b1bca> Trace;f88271ee<__crc_xfrm_state_get_afinfo+700ce/b1bca> Trace;c011b34e<__wake_up+e/20>
Trace;f8821c0c<__crc_xfrm_state_get_afinfo+6aaec/b1bca> Trace;f8858ad7<__crc_xfrm_state_get_afinfo+a19b7/b1bca> Trace;f8858a30<__crc_xfrm_state_get_afinfo+a1910/b1bca> Trace;c0170b3c<generic_delete_inode+8c/120> Trace;c016f93d<iput+4d/70>
Trace;c0168199<sys_unlink+159/1a0>
Trace;c0107dc9<sysenter_past_esp+52/79>Code;
c0126c38<__mod_timer+38/70>
00000000 <_EIP>:
Code;c0126c38<__mod_timer+38/70> <=====
0: 89 02 mov %eax,(%edx) <=====
Code;c0126c3a<__mod_timer+3a/70>
2: c7 43 04 00 02 20 00 movl $0x200200,0x4(%ebx)
Code;c0126c41<__mod_timer+41/70>
9: c7 03 00 01 10 00 movl $0x100100,(%ebx)
Code;c0126c47<__mod_timer+47/70>
f: 89 7b 08 mov %edi,0x8(%ebx)
Code;c0126c4a<__mod_timer+4a/70>
12: 89 da mov %ebx,%edx
Quoted 2 lines
kernel BUG at
fs/ext3/super.c:412!
invalid operand: 0000 [#2]
This is also a list related issue:
if (!list_empty(&sbi->s_orphan))
dump_orphan_list(sb, sbi);
J_ASSERT(list_empty(&sbi->s_orphan));Signature · 1 line
Jens Axboe
Hello Jens,
Thanks for your reply.
Quoted 1 line
Just to be on the safe side, have you reproduced it without?
Nope, I would first like to backup the data since this trace got me pretty scared and I care about the data on this disk. So I bought a DVD-R burner and I'm now happily doing my 160GB of backup.
Quoted 3 lines
Just
because nvidia doesn't show up in the trace, doesn't mean it hasn't corrupted memory elsewhere.
Fair enough.
Quoted 4 lines
00200200 is the list deletion poison, double remove of a list and in this case the timer base. That's pretty bad news.
So please do try and reproduce without the nvidia module.
I will try, for sure and I'll post back my results if I can make it oops again.. However I don't know how to reproduce it. I showed up once or twice during high I/O and while I was writing to and reading from disk over ieee1394. Could it also be a bad cable? The I/O is also dog slow when writing to disk, reading is like 15 MBytes/s.
Quoted 5 lines
This is also a list related issue:
if (!list_empty(&sbi->s_orphan)) dump_orphan_list(sb, sbi); J_ASSERT(list_empty(&sbi->s_orphan));
I'll also try a new firewire cable, since most of the times (in my experience) the cable has been the source of all problems. Normally the disk gets only remounted RO during deletion of files (this is a really irritating thing right now, I can't delete my files over ieee1394) which seems to prevent this kind of oops.
Thanks and best regards,
Roberto Nibali, ratz
Signature · 1 line
echo '[q]sa[ln0=aln256%Pln256/snlbx]sb3135071790101768542287578439snlbxq' | dc