Skip to contentSkip to conversation

ext3 bug

DISCUSSION Jean-Marc Valin 5 participants 14 messages Feb 27, 2005 – Mar 1, 2005

lore
Share & tools
lore
#0 lore raw

Hi,

Looks like I ran into an ext3 bug (or at least the log says so). I got a bunch of messages like:
ext3_free_blocks_sb: aborting transaction: Journal has aborted in __ext3_journal_get_undo_access<2>EXT3-fs error (device sda2) in
ext3_free_blocks_sb: Journal has aborted
EXT3-fs error (device sda2): ext3_free_blocks: Freeing blocks in system zones -Block = 228, count = 1

It happened while I was doing an "rm -rf" on a directory. The "rm" gave a segfault and now I can't unmount the filesystem: unmount says "device is busy", even though lsof reports nothing. The filesystem is on a USB hard disk. The actual dump is in attachment. I'm running Debian unstable with a custom 2.6.10 kernel on a 1.6 GHz Pentium-M.

Jean-Marc

Signature · 2 lines
Jean-Marc Valin <[email protected]>
Université de Sherbrooke
Feb 27 01:15:48 idefix kernel: ------------[ cut here ]------------
Feb 27 01:15:48 idefix kernel: PREEMPT 
Feb 27 01:15:48 idefix kernel: Modules linked in: msdos sd_mod udf isofs sr_mod usb_storage scsi_mod joydev usbhid appletalk ax25 ipx radeon ipt_state iptable_filter iptable_mangle iptable_nat ip_conntrack ip_tables ipv6 orinoco_cs orinoco hermes pcmcia lp binfmt_misc af_packet parport_pc parport uhci_hcd pci_hotplug intel_agp agpgart yenta_socket pcmcia_core tg3 snd_intel8x0 snd_ac97_codec ehci_hcd usbcore nls_iso8859_1 nls_cp437 vfat fat ppp_async ppp_generic slhc crc_ccitt snd_pcm_oss tsdev evdev snd_pcm snd_timer snd_page_alloc snd_mixer_oss snd soundcore psmouse thermal fan button ac battery cpufreq_ondemand cpufreq_powersave speedstep_centrino freq_table processor
Feb 27 01:15:48 idefix kernel: CPU:    0
Feb 27 01:15:48 idefix kernel: EIP:    0060:[<b01af540>]    Not tainted VLI
Feb 27 01:15:48 idefix kernel: EFLAGS: 00210286   (2.6.10) 
Feb 27 01:15:48 idefix kernel: EIP is at journal_forget+0x1d0/0x220
Feb 27 01:15:48 idefix kernel: eax: 0000005f   ebx: d1f1c000   ecx: b032c7cc   edx: b032c7cc
Feb 27 01:15:48 idefix kernel: esi: b8932d48   edi: bb2ad41c   ebp: dd668080   esp: d1f1dda0
Feb 27 01:15:48 idefix kernel: ds: 007b   es: 007b   ss: 0068
Feb 27 01:15:48 idefix kernel: Process rm (pid: 10370, threadinfo=d1f1c000 task=c97f49e0)
Feb 27 01:15:48 idefix kernel: Stack: b02f67e0 b02e1027 b02f445b 000004ca b02f4571 00000000 be0a5aac b8932d48 
Feb 27 01:15:48 idefix kernel:        dfc002b8 b019c940 dfc002b8 b8932d48 e73d7980 b275f400 b8932d48 00000006 
Feb 27 01:15:48 idefix kernel:        b0aeb448 dfc002b8 be0a5aac b019f028 dfc002b8 00000000 be0a5aac b8932d48 
Feb 27 01:15:48 idefix kernel: Call Trace:
Feb 27 01:15:48 idefix kernel:  [<b019c940>] ext3_forget+0xf0/0x100
Feb 27 01:15:48 idefix kernel:  [<b019f028>] ext3_clear_blocks+0x118/0x170
Feb 27 01:15:48 idefix kernel:  [<b019f118>] ext3_free_data+0x98/0x150
Feb 27 01:15:48 idefix kernel:  [<b019f2bc>] ext3_free_branches+0xec/0x270
Feb 27 01:15:48 idefix kernel:  [<b019f8ab>] ext3_truncate+0x46b/0x5d0
Feb 27 01:15:48 idefix kernel:  [<b01a08b8>] ext3_mark_iloc_dirty+0x28/0x40
Feb 27 01:15:48 idefix kernel:  [<b01ae12d>] journal_start+0xad/0xe0
Feb 27 01:15:48 idefix kernel:  [<b01a5234>] __ext3_journal_stop+0x24/0x50
Feb 27 01:15:48 idefix kernel:  [<b019c9a9>] start_transaction+0x29/0x70
Feb 27 01:15:48 idefix kernel:  [<b019cb28>] ext3_delete_inode+0xc8/0x100
Feb 27 01:15:48 idefix kernel:  [<b019ca60>] ext3_delete_inode+0x0/0x100
Feb 27 01:15:48 idefix kernel:  [<b01726f5>] generic_delete_inode+0xa5/0x170
Feb 27 01:15:48 idefix kernel:  [<b01729a3>] iput+0x63/0x90
Feb 27 01:15:48 idefix kernel:  [<b0167f27>] sys_unlink+0xd7/0x150
Feb 27 01:15:48 idefix kernel:  [<b016ad40>] sys_getdents64+0xa0/0xaa
Feb 27 01:15:48 idefix kernel:  [<b016aba0>] filldir64+0x0/0x100
Feb 27 01:15:48 idefix kernel:  [<b01030df>] syscall_call+0x7/0xb
Feb 27 01:15:48 idefix kernel: Code: 2f b0 b8 71 45 2f b0 89 44 24 10 b8 ca 04 00 00 89 44 24 0c b8 5b 44 2f b0 89 44 24 08 b8 27 10 2e b0 89 44 24 04 e8 c0 a6 f6 ff <0f> 0b ca 04 5b 44 2f b0 e9 4d ff ff ff c7 04 24 e0 67 2f b0 b8 
Feb 27 01:15:48 idefix kernel:  <6>note: rm[10370] exited with preempt_count 2
Feb 27 01:15:48 idefix kernel:  [<b02d8772>] schedule+0x532/0x540
Feb 27 01:15:48 idefix kernel:  [<b0146c53>] unmap_page_range+0x53/0x80
Feb 27 01:15:48 idefix kernel:  [<b0146e36>] unmap_vmas+0x1b6/0x1d0
Feb 27 01:15:48 idefix kernel:  [<b014b53d>] exit_mmap+0x7d/0x160
Feb 27 01:15:48 idefix kernel:  [<b0117617>] mmput+0x37/0xa0
Feb 27 01:15:48 idefix kernel:  [<b011c06f>] do_exit+0x16f/0x470
Feb 27 01:15:48 idefix kernel:  [<b01046a0>] do_invalid_op+0x0/0xd0
Feb 27 01:15:48 idefix kernel:  [<b01042cb>] die+0x18b/0x190
Feb 27 01:15:48 idefix kernel:  [<b0104752>] do_invalid_op+0xb2/0xd0
Feb 27 01:15:48 idefix kernel:  [<b01af540>] journal_forget+0x1d0/0x220
Feb 27 01:15:48 idefix kernel:  [<b01162d1>] __wake_up_common+0x41/0x70
Feb 27 01:15:48 idefix kernel:  [<b0119e9f>] release_console_sem+0xbf/0xd0
Feb 27 01:15:48 idefix kernel:  [<b0103b17>] error_code+0x2b/0x30
Feb 27 01:15:48 idefix kernel:  [<b01af540>] journal_forget+0x1d0/0x220
Feb 27 01:15:48 idefix kernel:  [<b019c940>] ext3_forget+0xf0/0x100
Feb 27 01:15:48 idefix kernel:  [<b019f028>] ext3_clear_blocks+0x118/0x170
Feb 27 01:15:48 idefix kernel:  [<b019f118>] ext3_free_data+0x98/0x150
Feb 27 01:15:48 idefix kernel:  [<b019f2bc>] ext3_free_branches+0xec/0x270
Feb 27 01:15:48 idefix kernel:  [<b019f8ab>] ext3_truncate+0x46b/0x5d0
Feb 27 01:15:48 idefix kernel:  [<b01a08b8>] ext3_mark_iloc_dirty+0x28/0x40
Feb 27 01:15:48 idefix kernel:  [<b01ae12d>] journal_start+0xad/0xe0
Feb 27 01:15:48 idefix kernel:  [<b01a5234>] __ext3_journal_stop+0x24/0x50
Feb 27 01:15:48 idefix kernel:  [<b019c9a9>] start_transaction+0x29/0x70
Feb 27 01:15:48 idefix kernel:  [<b019cb28>] ext3_delete_inode+0xc8/0x100
Feb 27 01:15:48 idefix kernel:  [<b019ca60>] ext3_delete_inode+0x0/0x100
Feb 27 01:15:48 idefix kernel:  [<b01726f5>] generic_delete_inode+0xa5/0x170
Feb 27 01:15:48 idefix kernel:  [<b01729a3>] iput+0x63/0x90
Feb 27 01:15:48 idefix kernel:  [<b0167f27>] sys_unlink+0xd7/0x150
Feb 27 01:15:48 idefix kernel:  [<b016ad40>] sys_getdents64+0xa0/0xaa
Feb 27 01:15:48 idefix kernel:  [<b016aba0>] filldir64+0x0/0x100
Feb 27 01:15:48 idefix kernel:  [<b01030df>] syscall_call+0x7/0xb
#1 lore raw
Quoted 10 lines · Jean-Marc Valin

Hi,

Looks like I ran into an ext3 bug (or at least the log says so). I got a bunch of messages like:
ext3_free_blocks_sb: aborting transaction: Journal has aborted in __ext3_journal_get_undo_access<2>EXT3-fs error (device sda2) in
ext3_free_blocks_sb: Journal has aborted
EXT3-fs error (device sda2): ext3_free_blocks: Freeing blocks in system zones -Block = 228, count = 1

Quoted 7 lines

It happened while I was doing an "rm -rf" on a directory. The "rm" gave a segfault and now I can't unmount the filesystem: unmount says "device is busy", even though lsof reports nothing. The filesystem is on a USB hard disk. The actual dump is in attachment. I'm running Debian unstable with a custom 2.6.10 kernel on a 1.6 GHz Pentium-M.

Jean-Marc

Please try stock kernel. 2.6.11-rc3 onwards should be fine. - I saw a similar problem while running 2.6.10 kernel from Fedora Core 3. It doesn't happen with stock kernels.

Parag

#2 lore raw
Quoted 17 lines · Parag Warudkar
Quoted 16 lines · Jean-Marc Valin

Hi,

Looks like I ran into an ext3 bug (or at least the log says so). I got a bunch of messages like:
ext3_free_blocks_sb: aborting transaction: Journal has aborted in __ext3_journal_get_undo_access<2>EXT3-fs error (device sda2) in
ext3_free_blocks_sb: Journal has aborted
EXT3-fs error (device sda2): ext3_free_blocks: Freeing blocks in system zones -Block = 228, count = 1

It happened while I was doing an "rm -rf" on a directory. The "rm" gave a segfault and now I can't unmount the filesystem: unmount says "device is busy", even though lsof reports nothing. The filesystem is on a USB hard disk. The actual dump is in attachment. I'm running Debian unstable with a custom 2.6.10 kernel on a 1.6 GHz Pentium-M.

Quoted 5 lines
Quoted 1 line

Jean-Marc

Please try stock kernel. 2.6.11-rc3 onwards should be fine. - I saw a similar problem while running 2.6.10 kernel from Fedora Core 3. It doesn't happen with stock kernels.

Which is very odd considering the only ext3 patches in the Fedora kernel are in 2.6.11rc.

Dave

#3 lore raw
Quoted 2 lines · Dave Jones

Which is very odd considering the only ext3 patches in the Fedora kernel are in 2.6.11rc.

This seems to be more of an USB-Storage issue than ext3.

#4 lore raw
In reply to Parag Warudkar
Quoted 3 lines

Please try stock kernel. 2.6.11-rc3 onwards should be fine. - I saw a similar problem while running 2.6.10 kernel from Fedora Core 3. It doesn't happen with stock kernels.

I did use a stock 2.6.10 kernel (I said custom in the sense that it wasn't a Debian kernel). After a reboot, I was able to run fsck on the disk (many, many errors) and it went fine after.

Jean-Marc

Signature · 2 lines
Jean-Marc Valin <[email protected]>
Université de Sherbrooke
#5 lore raw
Quoted 3 lines · Jean-Marc Valin

I did use a stock 2.6.10 kernel (I said custom in the sense that it wasn't a Debian kernel). After a reboot, I was able to run fsck on the disk (many, many errors) and it went fine after.

Hmm.. So that error is not FC3 specific, it is present in stock 2.6.10 as well. Also - This is on a USB disk, right? If so, the error may re-surface. Try upgrading to latest kernel if possible.

Parag

#6 lore raw
Quoted 3 lines

Hmm.. So that error is not FC3 specific, it is present in stock 2.6.10 as well. Also - This is on a USB disk, right? If so, the error may re-surface. Try upgrading to latest kernel if possible.

It's a USB disk (3.5" IDE + IDE to USB). What has been changed in 2.6.11-rcX?

Jean-Marc

Signature · 2 lines
Jean-Marc Valin <[email protected]>
Université de Sherbrooke
#7 lore raw
Quoted 2 lines · Jean-Marc Valin

What has been changed in
2.6.11-rcX?

Don't know exactly what changed but when I faced this issue Greg KH suggested to reproduce on 2.6.11-rc3 and it didn't happen there.

(I am assuming that the issue you are seeing is the same as what I saw since the error message looks similar and even I was using a USB disk when the error happened.)

Parag

#8 lore raw
In reply to Jean-Marc Valin

I see this problem infrequently on systems that have low memory conditions and
with heavy swapping. I have not seen it on 2.6.9 but I have seen it on 2.6.10.

Jeff

Quoted 12 lines · Jean-Marc Valin
Quoted 3 lines

Please try stock kernel. 2.6.11-rc3 onwards should be fine. - I saw a similar problem while running 2.6.10 kernel from Fedora Core 3. It doesn't happen with stock kernels.

I did use a stock 2.6.10 kernel (I said custom in the sense that it wasn't a Debian kernel). After a reboot, I was able to run fsck on the disk (many, many errors) and it went fine after.

Jean-Marc

#9 lore raw
Quoted 4 lines · jmerkey

I see this problem infrequently on systems that have low memory conditions and
with heavy swapping. I have not seen it on 2.6.9 but I have seen it
on 2.6.10.

My machine has 1 GB RAM and I wasn't using much of it at that time (2GB free on the swap), so I doubt that's the problem in my case.

Jean-Marc

Signature · 2 lines
Jean-Marc Valin <[email protected]>
Université de Sherbrooke
#10 lore raw
Quoted 8 lines · Jean-Marc Valin
Quoted 4 lines · jmerkey

I see this problem infrequently on systems that have low memory conditions and
with heavy swapping. I have not seen it on 2.6.9 but I have seen it on 2.6.10.

Quoted 7 lines

My machine has 1 GB RAM and I wasn't using much of it at that time (2GB free on the swap), so I doubt that's the problem in my case.

Jean-Marc

Running the ext2 recover program seems to trigger some good bugs in 2.6.10 with ext3 -- try it. I was doing this to test some disk tools and I managed to cause these errors with forcing ext2 recovery from an ext3 fs (which is probably something to be expected. The recover tools need to get syncrhonized -- have not tried with
mc yet.) Doesn't happen every time though.

Jeff

#11 lore raw
Quoted 16 lines · jmerkey
Quoted 14 lines · Jean-Marc Valin
Quoted 2 lines · jmerkey

I see this problem infrequently on systems that have low memory conditions and
with heavy swapping. I have not seen it on 2.6.9 but I have seen it on 2.6.10.

My machine has 1 GB RAM and I wasn't using much of it at that time (2GB free on the swap), so I doubt that's the problem in my case.

Jean-Marc

Quoted 9 lines

Running the ext2 recover program seems to trigger some good bugs in 2.6.10 with ext3 -- try it. I was doing this to test some disk tools and I managed to cause these errors with forcing ext2 recovery from an ext3 fs (which is probably something to be expected. The recover tools need to get syncrhonized -- have not tried with
mc yet.) Doesn't happen every time though.

Jeff

lde also causes some problems as well with ext3. Just caused one on 2.6.10. stale or poisoned cache blocks perhaps?

Jeff

#12 lore raw

linux-2.6.10 has some bio problems that are fixed in the current linux-2.6.11 release candidates. The bio problems wreaked havoc with XFS and there were people reporting EXT3 problems as well with this bug. I'd recommend trying the latest release candidate and see if your problem vanishes.

Signature · 1 line
jeffrey hundstad
Quoted 38 lines · jmerkey
Quoted 17 lines · jmerkey
Quoted 8 lines · Jean-Marc Valin
Quoted 2 lines · jmerkey

I see this problem infrequently on systems that have low memory conditions and
with heavy swapping. I have not seen it on 2.6.9 but I have seen it on 2.6.10.

Quoted 7 lines

My machine has 1 GB RAM and I wasn't using much of it at that time (2GB free on the swap), so I doubt that's the problem in my case.

Jean-Marc

Quoted 9 lines

Running the ext2 recover program seems to trigger some good bugs in 2.6.10 with ext3 -- try it. I was doing this to test some disk tools and I managed to cause these errors with forcing ext2 recovery from an ext3 fs (which is probably something to be expected. The recover tools need to get syncrhonized -- have not tried with
mc yet.) Doesn't happen every time though.

Jeff

lde also causes some problems as well with ext3. Just caused one on 2.6.10. stale or poisoned cache blocks perhaps?

Jeff
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#13 lore raw
Quoted 2 lines · Jeffrey E. Hundstad

linux-2.6.10 has some bio problems that are fixed in the current linux-2.6.11 release candidates. The bio problems wreaked havoc with XFS and there were people reporting EXT3 problems as well with this bug. I'd recommend trying the latest release candidate and see if your problem vanishes.

OK

Jeff