ext3 bug
Hi,
Looks like I ran into an ext3 bug (or at least the log says so). I got a bunch of messages like:ext3_free_blocks_sb: aborting transaction: Journal has aborted in __ext3_journal_get_undo_access<2>EXT3-fs error (device sda2) inext3_free_blocks_sb: Journal has aborted
EXT3-fs error (device sda2): ext3_free_blocks: Freeing blocks in system zones -Block = 228, count = 1
It happened while I was doing an "rm -rf" on a directory. The "rm" gave a segfault and now I can't unmount the filesystem: unmount says "device is busy", even though lsof reports nothing. The filesystem is on a USB hard disk. The actual dump is in attachment. I'm running Debian unstable with a custom 2.6.10 kernel on a 1.6 GHz Pentium-M.
Jean-Marc
Signature · 2 lines
Jean-Marc Valin <[email protected]> Université de Sherbrooke
Feb 27 01:15:48 idefix kernel: ------------[ cut here ]------------ Feb 27 01:15:48 idefix kernel: PREEMPT Feb 27 01:15:48 idefix kernel: Modules linked in: msdos sd_mod udf isofs sr_mod usb_storage scsi_mod joydev usbhid appletalk ax25 ipx radeon ipt_state iptable_filter iptable_mangle iptable_nat ip_conntrack ip_tables ipv6 orinoco_cs orinoco hermes pcmcia lp binfmt_misc af_packet parport_pc parport uhci_hcd pci_hotplug intel_agp agpgart yenta_socket pcmcia_core tg3 snd_intel8x0 snd_ac97_codec ehci_hcd usbcore nls_iso8859_1 nls_cp437 vfat fat ppp_async ppp_generic slhc crc_ccitt snd_pcm_oss tsdev evdev snd_pcm snd_timer snd_page_alloc snd_mixer_oss snd soundcore psmouse thermal fan button ac battery cpufreq_ondemand cpufreq_powersave speedstep_centrino freq_table processor Feb 27 01:15:48 idefix kernel: CPU: 0 Feb 27 01:15:48 idefix kernel: EIP: 0060:[<b01af540>] Not tainted VLI Feb 27 01:15:48 idefix kernel: EFLAGS: 00210286 (2.6.10) Feb 27 01:15:48 idefix kernel: EIP is at journal_forget+0x1d0/0x220 Feb 27 01:15:48 idefix kernel: eax: 0000005f ebx: d1f1c000 ecx: b032c7cc edx: b032c7cc Feb 27 01:15:48 idefix kernel: esi: b8932d48 edi: bb2ad41c ebp: dd668080 esp: d1f1dda0 Feb 27 01:15:48 idefix kernel: ds: 007b es: 007b ss: 0068 Feb 27 01:15:48 idefix kernel: Process rm (pid: 10370, threadinfo=d1f1c000 task=c97f49e0) Feb 27 01:15:48 idefix kernel: Stack: b02f67e0 b02e1027 b02f445b 000004ca b02f4571 00000000 be0a5aac b8932d48 Feb 27 01:15:48 idefix kernel: dfc002b8 b019c940 dfc002b8 b8932d48 e73d7980 b275f400 b8932d48 00000006 Feb 27 01:15:48 idefix kernel: b0aeb448 dfc002b8 be0a5aac b019f028 dfc002b8 00000000 be0a5aac b8932d48 Feb 27 01:15:48 idefix kernel: Call Trace: Feb 27 01:15:48 idefix kernel: [<b019c940>] ext3_forget+0xf0/0x100 Feb 27 01:15:48 idefix kernel: [<b019f028>] ext3_clear_blocks+0x118/0x170 Feb 27 01:15:48 idefix kernel: [<b019f118>] ext3_free_data+0x98/0x150 Feb 27 01:15:48 idefix kernel: [<b019f2bc>] ext3_free_branches+0xec/0x270 Feb 27 01:15:48 idefix kernel: [<b019f8ab>] ext3_truncate+0x46b/0x5d0 Feb 27 01:15:48 idefix kernel: [<b01a08b8>] ext3_mark_iloc_dirty+0x28/0x40 Feb 27 01:15:48 idefix kernel: [<b01ae12d>] journal_start+0xad/0xe0 Feb 27 01:15:48 idefix kernel: [<b01a5234>] __ext3_journal_stop+0x24/0x50 Feb 27 01:15:48 idefix kernel: [<b019c9a9>] start_transaction+0x29/0x70 Feb 27 01:15:48 idefix kernel: [<b019cb28>] ext3_delete_inode+0xc8/0x100 Feb 27 01:15:48 idefix kernel: [<b019ca60>] ext3_delete_inode+0x0/0x100 Feb 27 01:15:48 idefix kernel: [<b01726f5>] generic_delete_inode+0xa5/0x170 Feb 27 01:15:48 idefix kernel: [<b01729a3>] iput+0x63/0x90 Feb 27 01:15:48 idefix kernel: [<b0167f27>] sys_unlink+0xd7/0x150 Feb 27 01:15:48 idefix kernel: [<b016ad40>] sys_getdents64+0xa0/0xaa Feb 27 01:15:48 idefix kernel: [<b016aba0>] filldir64+0x0/0x100 Feb 27 01:15:48 idefix kernel: [<b01030df>] syscall_call+0x7/0xb Feb 27 01:15:48 idefix kernel: Code: 2f b0 b8 71 45 2f b0 89 44 24 10 b8 ca 04 00 00 89 44 24 0c b8 5b 44 2f b0 89 44 24 08 b8 27 10 2e b0 89 44 24 04 e8 c0 a6 f6 ff <0f> 0b ca 04 5b 44 2f b0 e9 4d ff ff ff c7 04 24 e0 67 2f b0 b8 Feb 27 01:15:48 idefix kernel: <6>note: rm[10370] exited with preempt_count 2 Feb 27 01:15:48 idefix kernel: [<b02d8772>] schedule+0x532/0x540 Feb 27 01:15:48 idefix kernel: [<b0146c53>] unmap_page_range+0x53/0x80 Feb 27 01:15:48 idefix kernel: [<b0146e36>] unmap_vmas+0x1b6/0x1d0 Feb 27 01:15:48 idefix kernel: [<b014b53d>] exit_mmap+0x7d/0x160 Feb 27 01:15:48 idefix kernel: [<b0117617>] mmput+0x37/0xa0 Feb 27 01:15:48 idefix kernel: [<b011c06f>] do_exit+0x16f/0x470 Feb 27 01:15:48 idefix kernel: [<b01046a0>] do_invalid_op+0x0/0xd0 Feb 27 01:15:48 idefix kernel: [<b01042cb>] die+0x18b/0x190 Feb 27 01:15:48 idefix kernel: [<b0104752>] do_invalid_op+0xb2/0xd0 Feb 27 01:15:48 idefix kernel: [<b01af540>] journal_forget+0x1d0/0x220 Feb 27 01:15:48 idefix kernel: [<b01162d1>] __wake_up_common+0x41/0x70 Feb 27 01:15:48 idefix kernel: [<b0119e9f>] release_console_sem+0xbf/0xd0 Feb 27 01:15:48 idefix kernel: [<b0103b17>] error_code+0x2b/0x30 Feb 27 01:15:48 idefix kernel: [<b01af540>] journal_forget+0x1d0/0x220 Feb 27 01:15:48 idefix kernel: [<b019c940>] ext3_forget+0xf0/0x100 Feb 27 01:15:48 idefix kernel: [<b019f028>] ext3_clear_blocks+0x118/0x170 Feb 27 01:15:48 idefix kernel: [<b019f118>] ext3_free_data+0x98/0x150 Feb 27 01:15:48 idefix kernel: [<b019f2bc>] ext3_free_branches+0xec/0x270 Feb 27 01:15:48 idefix kernel: [<b019f8ab>] ext3_truncate+0x46b/0x5d0 Feb 27 01:15:48 idefix kernel: [<b01a08b8>] ext3_mark_iloc_dirty+0x28/0x40 Feb 27 01:15:48 idefix kernel: [<b01ae12d>] journal_start+0xad/0xe0 Feb 27 01:15:48 idefix kernel: [<b01a5234>] __ext3_journal_stop+0x24/0x50 Feb 27 01:15:48 idefix kernel: [<b019c9a9>] start_transaction+0x29/0x70 Feb 27 01:15:48 idefix kernel: [<b019cb28>] ext3_delete_inode+0xc8/0x100 Feb 27 01:15:48 idefix kernel: [<b019ca60>] ext3_delete_inode+0x0/0x100 Feb 27 01:15:48 idefix kernel: [<b01726f5>] generic_delete_inode+0xa5/0x170 Feb 27 01:15:48 idefix kernel: [<b01729a3>] iput+0x63/0x90 Feb 27 01:15:48 idefix kernel: [<b0167f27>] sys_unlink+0xd7/0x150 Feb 27 01:15:48 idefix kernel: [<b016ad40>] sys_getdents64+0xa0/0xaa Feb 27 01:15:48 idefix kernel: [<b016aba0>] filldir64+0x0/0x100 Feb 27 01:15:48 idefix kernel: [<b01030df>] syscall_call+0x7/0xb
Quoted 10 lines · Jean-Marc Valin
Hi,
Looks like I ran into an ext3 bug (or at least the log says so). I got a bunch of messages like:
ext3_free_blocks_sb: aborting transaction: Journal has aborted in __ext3_journal_get_undo_access<2>EXT3-fs error (device sda2) inext3_free_blocks_sb: Journal has aborted
EXT3-fs error (device sda2):ext3_free_blocks: Freeing blocks in system zones -Block = 228, count = 1
Quoted 7 lines
It happened while I was doing an "rm -rf" on a directory. The "rm" gave a segfault and now I can't unmount the filesystem: unmount says "device is busy", even though lsof reports nothing. The filesystem is on a USB hard disk. The actual dump is in attachment. I'm running Debian unstable with a custom 2.6.10 kernel on a 1.6 GHz Pentium-M.
Jean-Marc
Please try stock kernel. 2.6.11-rc3 onwards should be fine. - I saw a similar problem while running 2.6.10 kernel from Fedora Core 3. It doesn't happen with stock kernels.
Parag
Quoted 17 lines · Parag Warudkar
Quoted 16 lines · Jean-Marc Valin
Hi,
Looks like I ran into an ext3 bug (or at least the log says so). I got a bunch of messages like:
ext3_free_blocks_sb: aborting transaction: Journal has aborted in __ext3_journal_get_undo_access<2>EXT3-fs error (device sda2) inext3_free_blocks_sb: Journal has aborted
EXT3-fs error (device sda2):ext3_free_blocks: Freeing blocks in system zones -Block = 228, count = 1It happened while I was doing an "rm -rf" on a directory. The "rm" gave a segfault and now I can't unmount the filesystem: unmount says "device is busy", even though lsof reports nothing. The filesystem is on a USB hard disk. The actual dump is in attachment. I'm running Debian unstable with a custom 2.6.10 kernel on a 1.6 GHz Pentium-M.
Quoted 5 lines
Quoted 1 line
Jean-Marc
Please try stock kernel. 2.6.11-rc3 onwards should be fine. - I saw a similar problem while running 2.6.10 kernel from Fedora Core 3. It doesn't happen with stock kernels.
Which is very odd considering the only ext3 patches in the Fedora kernel are in 2.6.11rc.
Dave
Quoted 2 lines · Dave Jones
Which is very odd considering the only ext3 patches in the Fedora kernel are in 2.6.11rc.
This seems to be more of an USB-Storage issue than ext3.
Quoted 3 lines
Please try stock kernel. 2.6.11-rc3 onwards should be fine. - I saw a similar problem while running 2.6.10 kernel from Fedora Core 3. It doesn't happen with stock kernels.
I did use a stock 2.6.10 kernel (I said custom in the sense that it wasn't a Debian kernel). After a reboot, I was able to run fsck on the disk (many, many errors) and it went fine after.
Jean-Marc
Signature · 2 lines
Jean-Marc Valin <[email protected]> Université de Sherbrooke
Quoted 3 lines · Jean-Marc Valin
I did use a stock 2.6.10 kernel (I said custom in the sense that it wasn't a Debian kernel). After a reboot, I was able to run fsck on the disk (many, many errors) and it went fine after.
Hmm.. So that error is not FC3 specific, it is present in stock 2.6.10 as well. Also - This is on a USB disk, right? If so, the error may re-surface. Try upgrading to latest kernel if possible.
Parag
Quoted 3 lines
Hmm.. So that error is not FC3 specific, it is present in stock 2.6.10 as well. Also - This is on a USB disk, right? If so, the error may re-surface. Try upgrading to latest kernel if possible.
It's a USB disk (3.5" IDE + IDE to USB). What has been changed in 2.6.11-rcX?
Jean-Marc
Signature · 2 lines
Jean-Marc Valin <[email protected]> Université de Sherbrooke
Quoted 2 lines · Jean-Marc Valin
What has been changed in
2.6.11-rcX?
Don't know exactly what changed but when I faced this issue Greg KH suggested to reproduce on 2.6.11-rc3 and it didn't happen there.
(I am assuming that the issue you are seeing is the same as what I saw since the error message looks similar and even I was using a USB disk when the error happened.)
Parag
I see this problem infrequently on systems that have low memory conditions and
with heavy swapping. I have not seen it on 2.6.9 but I have seen it on 2.6.10.
Jeff
Quoted 12 lines · Jean-Marc Valin
Quoted 3 lines
Please try stock kernel. 2.6.11-rc3 onwards should be fine. - I saw a similar problem while running 2.6.10 kernel from Fedora Core 3. It doesn't happen with stock kernels.
I did use a stock 2.6.10 kernel (I said custom in the sense that it wasn't a Debian kernel). After a reboot, I was able to run fsck on the disk (many, many errors) and it went fine after.
Jean-Marc
Quoted 4 lines · jmerkey
I see this problem infrequently on systems that have low memory conditions and
with heavy swapping. I have not seen it on 2.6.9 but I have seen it
on 2.6.10.
My machine has 1 GB RAM and I wasn't using much of it at that time (2GB free on the swap), so I doubt that's the problem in my case.
Jean-Marc
Signature · 2 lines
Jean-Marc Valin <[email protected]> Université de Sherbrooke
Quoted 8 lines · Jean-Marc Valin
Quoted 4 lines · jmerkey
I see this problem infrequently on systems that have low memory conditions and
with heavy swapping. I have not seen it on 2.6.9 but I have seen it on 2.6.10.
Quoted 7 lines
My machine has 1 GB RAM and I wasn't using much of it at that time (2GB free on the swap), so I doubt that's the problem in my case.
Jean-Marc
Running the ext2 recover program seems to trigger some good bugs in 2.6.10 with ext3 -- try it. I was doing this to test some disk tools and I managed to cause these errors with forcing ext2 recovery from an ext3 fs (which is probably something to be expected. The recover tools need to get syncrhonized -- have not tried with
mc yet.) Doesn't happen every time though.
Jeff
Quoted 16 lines · jmerkey
Quoted 14 lines · Jean-Marc Valin
Quoted 2 lines · jmerkey
I see this problem infrequently on systems that have low memory conditions and
with heavy swapping. I have not seen it on 2.6.9 but I have seen it on 2.6.10.My machine has 1 GB RAM and I wasn't using much of it at that time (2GB free on the swap), so I doubt that's the problem in my case.
Jean-Marc
Quoted 9 lines
Running the ext2 recover program seems to trigger some good bugs in 2.6.10 with ext3 -- try it. I was doing this to test some disk tools and I managed to cause these errors with forcing ext2 recovery from an ext3 fs (which is probably something to be expected. The recover tools need to get syncrhonized -- have not tried with
mc yet.) Doesn't happen every time though.Jeff
lde also causes some problems as well with ext3. Just caused one on 2.6.10. stale or poisoned cache blocks perhaps?
Jeff
linux-2.6.10 has some bio problems that are fixed in the current linux-2.6.11 release candidates. The bio problems wreaked havoc with XFS and there were people reporting EXT3 problems as well with this bug. I'd recommend trying the latest release candidate and see if your problem vanishes.
Signature · 1 line
jeffrey hundstad
Quoted 38 lines · jmerkey
Quoted 17 lines · jmerkey
Quoted 8 lines · Jean-Marc Valin
Quoted 2 lines · jmerkey
I see this problem infrequently on systems that have low memory conditions and
with heavy swapping. I have not seen it on 2.6.9 but I have seen it on 2.6.10.Quoted 7 lines
My machine has 1 GB RAM and I wasn't using much of it at that time (2GB free on the swap), so I doubt that's the problem in my case.
Jean-Marc
Quoted 9 lines
Running the ext2 recover program seems to trigger some good bugs in 2.6.10 with ext3 -- try it. I was doing this to test some disk tools and I managed to cause these errors with forcing ext2 recovery from an ext3 fs (which is probably something to be expected. The recover tools need to get syncrhonized -- have not tried with
mc yet.) Doesn't happen every time though.Jeff
lde also causes some problems as well with ext3. Just caused one on 2.6.10. stale or poisoned cache blocks perhaps?
Jeff
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 2 lines · Jeffrey E. Hundstad
linux-2.6.10 has some bio problems that are fixed in the current linux-2.6.11 release candidates. The bio problems wreaked havoc with XFS and there were people reporting EXT3 problems as well with this bug. I'd recommend trying the latest release candidate and see if your problem vanishes.
OK
Jeff