oops in ext3_discard_reservation()
Greetings,
I just received the oops below. This kernel is mostly virgin, but does contain a small bundle of scheduler modifications. Salt to taste.
-Mike
BUG: unable to handle kernel paging request at virtual address 0001001c printing eip: c10ab599 *pde = 00000000 Oops: 0000 [#1] PREEMPT SMP last sysfs file: /devices/system/cpu/cpu0/cpufreq/scaling_setspeed Modules linked in: ohci1394 prism54 xt_tcpudp xt_pkttype ipt_LOG xt_limit snd_pcm_oss snd_mixer_oss snd_seq snd_seq_device edd tda9887 saa7134 snd_intel8x0 snd_ac97_codec ieee1394 snd_ac97_bus bt878 snd_pcm ir_kbd_i2c snd_timer snd soundcore i2c_i801 snd_page_alloc ipt_REJECT xt_state iptable_mangle iptable_nat ip_nat iptable_filter ip6table_mangle ip6table_filter ip_conntrack ip_tables ip6_tables x_tables tuner bttv video_buf firmware_class ir_common btcx_risc tveeprom sd_mod nls_iso8859_1 nls_cp437 nls_utf8 CPU: 1 EIP: 0060:[<c10ab599>] Not tainted VLI EFLAGS: 00210206 (2.6.16-rc6-mm1x-smp #19)
EIP is at ext3_discard_reservation+0x29/0x63
eax: f7e39100 ebx: c1583b50 ecx: 00000000 edx: c10b9ca7
esi: 00010000 edi: 00010018 ebp: dffc2e9c esp: dffc2e8c
ds: 007b es: 007b ss: 0068
Process kswapd0 (pid: 583, threadinfo=dffc2000 task=dffc6030)
Stack: <0>f7e39100 ffffffff c1583ab0 c1583b50 dffc2eb8 c10b9d0b dffc2eb8 00010000
c1583b50 c10b9dc1 dffc2ef8 dffc2ecc c10835e5 dffc2ecc c1583b58 c1583b50
dffc2ee4 c1083669 00000012 00000080 f71817a8 f71817a0 dffc2f0c c1083915
Call Trace:
<c1004073> show_stack_log_lvl+0xa9/0xe3 <c100424d> show_registers+0x1a0/0x236
<c1004561> die+0x12f/0x2ae <c1019aff> do_page_fault+0x353/0x5fa
<c1003a2b> error_code+0x4f/0x54 <c10b9d0b> ext3_clear_inode+0x64/0xb6
<c10835e5> clear_inode+0xa3/0x103 <c1083669> dispose_list+0x24/0x103
<c1083915> shrink_icache_memory+0x1cd/0x220 <c10533c7> shrink_slab+0x160/0x1f9
<c1054942> balance_pgdat+0x200/0x3be <c1054bd9> kswapd+0xd9/0x125
<c1001005> kernel_thread_helper+0x5/0xb
Code: 5d c3 55 89 e5 57 56 53 83 ec 04 89 c3 8b 70 b4 8b 80 a8 00 00 00 8b 80 78 01 00 00 05 00 11 00 00 89 45 f0 85 f6 74 0a 8d 7e 18 <8b> 57 04 85 d2 75 08 83 c4 04 5b 5e 5f 5d c3 e8 2e 41 30 00 8bQuoted 26 lines · Mike Galbraith
I just received the oops below. This kernel is mostly virgin, but does
contain a small bundle of scheduler modifications. Salt to taste.-Mike
BUG: unable to handle kernel paging request at virtual address 0001001c printing eip: c10ab599 *pde = 00000000 Oops: 0000 [#1] PREEMPT SMP last sysfs file: /devices/system/cpu/cpu0/cpufreq/scaling_setspeed Modules linked in: ohci1394 prism54 xt_tcpudp xt_pkttype ipt_LOG xt_limit snd_pcm_oss snd_mixer_oss snd_seq snd_seq_device edd tda9887 saa7134 snd_intel8x0 snd_ac97_codec ieee1394 snd_ac97_bus bt878 snd_pcm ir_kbd_i2c snd_timer snd soundcore i2c_i801 snd_page_alloc ipt_REJECT xt_state iptable_mangle iptable_nat ip_nat iptable_filter ip6table_mangle ip6table_filter ip_conntrack ip_tables ip6_tables x_tables tuner bttv video_buf firmware_class ir_common btcx_risc tveeprom sd_mod nls_iso8859_1 nls_cp437 nls_utf8 CPU: 1 EIP: 0060:[<c10ab599>] Not tainted VLI EFLAGS: 00210206 (2.6.16-rc6-mm1x-smp #19) EIP is at ext3_discard_reservation+0x29/0x63 eax: f7e39100 ebx: c1583b50 ecx: 00000000 edx: c10b9ca7 esi: 00010000 edi: 00010018 ebp: dffc2e9c esp: dffc2e8c ds: 007b es: 007b ss: 0068 Process kswapd0 (pid: 583, threadinfo=dffc2000 task=dffc6030) Stack: <0>f7e39100 ffffffff c1583ab0 c1583b50 dffc2eb8 c10b9d0b dffc2eb8 00010000 c1583b50 c10b9dc1 dffc2ef8 dffc2ecc c10835e5 dffc2ecc c1583b58 c1583b50 dffc2ee4 c1083669 00000012 00000080 f71817a8 f71817a0 dffc2f0c c1083915 Call Trace:
Quoted 8 lines
<c1004073> show_stack_log_lvl+0xa9/0xe3 <c100424d> show_registers+0x1a0/0x236 <c1004561> die+0x12f/0x2ae <c1019aff> do_page_fault+0x353/0x5fa <c1003a2b> error_code+0x4f/0x54 <c10b9d0b> ext3_clear_inode+0x64/0xb6 <c10835e5> clear_inode+0xa3/0x103 <c1083669> dispose_list+0x24/0x103 <c1083915> shrink_icache_memory+0x1cd/0x220 <c10533c7> shrink_slab+0x160/0x1f9 <c1054942> balance_pgdat+0x200/0x3be <c1054bd9> kswapd+0xd9/0x125 <c1001005> kernel_thread_helper+0x5/0xb Code: 5d c3 55 89 e5 57 56 53 83 ec 04 89 c3 8b 70 b4 8b 80 a8 00 00 00 8b 80 78 01 00 00 05 00 11 00 00 89 45 f0 85 f6 74 0a 8d 7e 18 <8b> 57 04 85 d2 75 08 83 c4 04 5b 5e 5f 5d c3 e8 2e 41 30 00 8b
It died in rsv_is_empty(), accessing rsv->_rsv_end, because `block_i' has a value of 0x00010000.
Looks like a bitflip. How good is that hardware?
Quoted 5 lines · Andrew Morton
It died in
rsv_is_empty(), accessingrsv->_rsv_end, because `block_i' has a value of0x00010000.Looks like a bitflip. How good is that hardware?
Solid as a rock to date fwtw.
-Mike
Quoted 36 lines · Andrew Morton
Quoted 26 lines · Mike Galbraith
I just received the oops below. This kernel is mostly virgin, but does
contain a small bundle of scheduler modifications. Salt to taste.-Mike
BUG: unable to handle kernel paging request at virtual address 0001001c printing eip: c10ab599 *pde = 00000000 Oops: 0000 [#1] PREEMPT SMP last sysfs file: /devices/system/cpu/cpu0/cpufreq/scaling_setspeed Modules linked in: ohci1394 prism54 xt_tcpudp xt_pkttype ipt_LOG xt_limit snd_pcm_oss snd_mixer_oss snd_seq snd_seq_device edd tda9887 saa7134 snd_intel8x0 snd_ac97_codec ieee1394 snd_ac97_bus bt878 snd_pcm ir_kbd_i2c snd_timer snd soundcore i2c_i801 snd_page_alloc ipt_REJECT xt_state iptable_mangle iptable_nat ip_nat iptable_filter ip6table_mangle ip6table_filter ip_conntrack ip_tables ip6_tables x_tables tuner bttv video_buf firmware_class ir_common btcx_risc tveeprom sd_mod nls_iso8859_1 nls_cp437 nls_utf8 CPU: 1 EIP: 0060:[<c10ab599>] Not tainted VLI EFLAGS: 00210206 (2.6.16-rc6-mm1x-smp #19) EIP is at ext3_discard_reservation+0x29/0x63 eax: f7e39100 ebx: c1583b50 ecx: 00000000 edx: c10b9ca7 esi: 00010000 edi: 00010018 ebp: dffc2e9c esp: dffc2e8c ds: 007b es: 007b ss: 0068 Process kswapd0 (pid: 583, threadinfo=dffc2000 task=dffc6030) Stack: <0>f7e39100 ffffffff c1583ab0 c1583b50 dffc2eb8 c10b9d0b dffc2eb8 00010000 c1583b50 c10b9dc1 dffc2ef8 dffc2ecc c10835e5 dffc2ecc c1583b58 c1583b50 dffc2ee4 c1083669 00000012 00000080 f71817a8 f71817a0 dffc2f0c c1083915 Call Trace:Quoted 8 lines
<c1004073> show_stack_log_lvl+0xa9/0xe3 <c100424d> show_registers+0x1a0/0x236 <c1004561> die+0x12f/0x2ae <c1019aff> do_page_fault+0x353/0x5fa <c1003a2b> error_code+0x4f/0x54 <c10b9d0b> ext3_clear_inode+0x64/0xb6 <c10835e5> clear_inode+0xa3/0x103 <c1083669> dispose_list+0x24/0x103 <c1083915> shrink_icache_memory+0x1cd/0x220 <c10533c7> shrink_slab+0x160/0x1f9 <c1054942> balance_pgdat+0x200/0x3be <c1054bd9> kswapd+0xd9/0x125 <c1001005> kernel_thread_helper+0x5/0xb Code: 5d c3 55 89 e5 57 56 53 83 ec 04 89 c3 8b 70 b4 8b 80 a8 00 00 00 8b 80 78 01 00 00 05 00 11 00 00 89 45 f0 85 f6 74 0a 8d 7e 18 <8b> 57 04 85 d2 75 08 83 c4 04 5b 5e 5f 5d c3 e8 2e 41 30 00 8b
Quoted 4 lines
It died in
rsv_is_empty(), accessingrsv->_rsv_end, because `block_i' has a value of0x00010000.Looks like a bitflip. How good is that hardware?
I am not sure what is happening. Smells like another race between two reservation window freeing. I thought we have thought this through quite well last year but maybe not.:(
In general, the reservation window allocating and freeing should be protected by a semaphore, so that we will not run into the case where we are trying to free a window, while another thread is in the middle of window-freed-and-then-re-allocating. But in the path of ext3_delete_inode()->clear_inode()->ext3_clear_inode()-
Quoted 1 line
ext3_discard_reservation(), since it's only called at the lastinput(),
so this race seems not exists.
But with the path of invalidate_inodes->dispose_list->clear_inode-
Quoted 1 line
ext3_discard_inode, I guess it is still possible that another thread is
in the middle of block allocation, who just freed the current window and is about to allocating a new window?
Mingming
Quoted 7 lines · Mingming Cao
Quoted 5 lines
It died in
rsv_is_empty(), accessingrsv->_rsv_end, because `block_i' has a value of0x00010000.Looks like a bitflip. How good is that hardware?
I am not sure what is happening.
Sorry, I cc'ed you, then worked out what happened then forgot to un-cc you. It looks like Mike's machine suffered a single-bit hardware error.
Quoted 11 lines
Smells like another race between two
reservation window freeing. I thought we have thought this through quite well last year but maybe not.:(In general, the reservation window allocating and freeing should be protected by a semaphore, so that we will not run into the case where we are trying to free a window, while another thread is in the middle of window-freed-and-then-re-allocating. But in the path of ext3_delete_inode()->clear_inode()->ext3_clear_inode()-
Quoted 1 line
ext3_discard_reservation(), since it's only called at the lastinput(),so this race seems not exists.
Yes, it looks OK.
Quoted 4 lines
But with the path of invalidate_inodes->dispose_list->clear_inode-
Quoted 1 line
ext3_discard_inode, I guess it is still possible that another thread isin the middle of block allocation, who just freed the current window and is about to allocating a new window?
The inode shootdown logic is quite gruesome, but we'd have heard about it if it was possible for block allocation to be happening while an inode has I_FREEING set. Various things in fs/inode.c would explode.