Ext3 mounted as ext2 but journal still in effect.
I have an ext3 filesystem that has been having problems with its journal. The result is that the file system remounts internally as read-only and the server becomes unusable, even shutdown does not work, using up 100% of the CPU but not rebooting.
I found some postings indicating that mounting it as ext2 should fix the problem, as it doesn't appear to be a hardware issue.
So, I decided to mount everything as ext2. Mount shows this:
# mount /dev/hda2 on / type ext2 (rw,usrquota) none on /proc type proc (rw) none on /sys type sysfs (rw) none on /dev/pts type devpts (rw,gid=5,mode=620) usbfs on /proc/bus/usb type usbfs (rw) /dev/hda1 on /boot type ext2 (rw) none on /dev/shm type tmpfs (rw,noexec) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
But now I still get the error:
# dmesg [...] EXT3-fs error (device hda2) in start_transaction: Journal has aborted EXT3-fs error (device hda2) in start_transaction: Journal has aborted EXT3-fs error (device hda2) in start_transaction: Journal has aborted EXT3-fs error (device hda2) in start_transaction: Journal has aborted [...]
The kernel is:
# uname -a Linux hyperweb.net 2.6.5-1.358smp #1 SMP Sat May 8 09:25:36 EDT 2004 i686 i686 i386 GNU/Linux
Any ideas?
Thanks,
Alberto
Signature · 3 lines
Alberto Alonso Global Gate Systems LLC. (512) 351-7233 http://www.ggsys.net Hardware, consulting, sysadmin, monitoring and remote backups
Quoted 34 lines · On Thu, 11 Jan 2007 23:08:16 -0600 Alberto Alonso <[email protected]> wrote:
I have an ext3 filesystem that has been having problems with its journal. The result is that the file system remounts internally as read-only and the server becomes unusable, even shutdown does not work, using up 100% of the CPU but not rebooting.
I found some postings indicating that mounting it as ext2 should fix the problem, as it doesn't appear to be a hardware issue.
So, I decided to mount everything as ext2. Mount shows this:
# mount /dev/hda2 on / type ext2 (rw,usrquota) none on /proc type proc (rw) none on /sys type sysfs (rw) none on /dev/pts type devpts (rw,gid=5,mode=620) usbfs on /proc/bus/usb type usbfs (rw) /dev/hda1 on /boot type ext2 (rw) none on /dev/shm type tmpfs (rw,noexec) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)But now I still get the error:
# dmesg [...] EXT3-fs error (device hda2) in start_transaction: Journal has aborted EXT3-fs error (device hda2) in start_transaction: Journal has aborted EXT3-fs error (device hda2) in start_transaction: Journal has aborted EXT3-fs error (device hda2) in start_transaction: Journal has aborted [...]
Quoted 9 lines
The kernel is:
# uname -a Linux hyperweb.net 2.6.5-1.358smp #1 SMP Sat May 8 09:25:36 EDT 2004 i686 i686 i386 GNU/LinuxAny ideas?
mount(8) tells lies. Look in /proc/mounts and you'll see that it's really mounted as ext3.
You probably want to add `rootfstype=ext2' to the kernel boot command line.
You were right, even after making the changes, it seems to be telling lies:
# mount /dev/hda2 on / type ext2 (rw,usrquota) [...]
However, I think I am still not mounting as ext2:
# dmesg | grep 'Kernel command' Kernel command line: ro root=/dev/hda2 rootfstype=ext2
# cat /proc/mounts rootfs / rootfs rw 0 0 /dev/root / ext3 rw 0 0 /proc /proc proc rw,nodiratime 0 0 /sys /sys sysfs rw 0 0 none /dev/pts devpts rw 0 0 usbfs /proc/bus/usb usbfs rw 0 0 /dev/hda1 /boot ext2 rw 0 0 none /dev/shm tmpfs rw,noexec 0 0 none /proc/sys/fs/binfmt_misc binfmt_misc rw 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw 0 0
Do I need to mess with the initrd? My grub lines look like this:
title Fedora Core (2.6.5-1.358smp)
root (hd0,0)
kernel /vmlinuz-2.6.5-1.358smp ro root=/dev/hda2 rootfstype=ext2
initrd /initrd-2.6.5-1.358smp.img
title Fedora Core-up (2.6.5-1.358)
root (hd0,0)
kernel /vmlinuz-2.6.5-1.358 ro root=/dev/hda2 rootfstype=ext2
initrd /initrd-2.6.5-1.358.imgThanks,
Alberto
Quoted 52 lines · Andrew Morton
Quoted 34 lines · On Thu, 11 Jan 2007 23:08:16 -0600 Alberto Alonso <[email protected]> wrote:
I have an ext3 filesystem that has been having problems with its journal. The result is that the file system remounts internally as read-only and the server becomes unusable, even shutdown does not work, using up 100% of the CPU but not rebooting.
I found some postings indicating that mounting it as ext2 should fix the problem, as it doesn't appear to be a hardware issue.
So, I decided to mount everything as ext2. Mount shows this:
# mount /dev/hda2 on / type ext2 (rw,usrquota) none on /proc type proc (rw) none on /sys type sysfs (rw) none on /dev/pts type devpts (rw,gid=5,mode=620) usbfs on /proc/bus/usb type usbfs (rw) /dev/hda1 on /boot type ext2 (rw) none on /dev/shm type tmpfs (rw,noexec) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)But now I still get the error:
# dmesg [...] EXT3-fs error (device hda2) in start_transaction: Journal has aborted EXT3-fs error (device hda2) in start_transaction: Journal has aborted EXT3-fs error (device hda2) in start_transaction: Journal has aborted EXT3-fs error (device hda2) in start_transaction: Journal has aborted [...]Quoted 9 lines
The kernel is:
# uname -a Linux hyperweb.net 2.6.5-1.358smp #1 SMP Sat May 8 09:25:36 EDT 2004 i686 i686 i386 GNU/LinuxAny ideas?
mount(8) tells lies. Look in
/proc/mountsand you'll see that it's really mounted as ext3.You probably want to add `rootfstype=ext2' to the kernel boot command line.
Signature · 3 lines
Alberto Alonso Global Gate Systems LLC. (512) 351-7233 http://www.ggsys.net Hardware, consulting, sysadmin, monitoring and remote backups
Hi!
Quoted 11 lines
You were right, even after making the changes, it seems to be telling lies:
# mount /dev/hda2 on / type ext2 (rw,usrquota) [...]However, I think I am still not mounting as ext2:
# dmesg | grep 'Kernel command' Kernel command line: ro root=/dev/hda2 rootfstype=ext2
...
Quoted 6 lines
rootfs / rootfs rw 0 0
/dev/root/ ext3 rw 0 0Do I need to mess with the initrd? My grub lines look like this:
Yes, probably.
Pavel
Signature · 1 line
Thanks for all the (sleeping) penguins.
Why not just use tune2fs to remove the ext3 journal?
Quoted 6 lines
Quoted 6 lines
You were right, even after making the changes, it seems to be telling lies:
# mount /dev/hda2 on / type ext2 (rw,usrquota)
Roughly speaking:/etc/mtab shows you what you said to mount. /proc/mounts shows what the current kernel state is. These may differ greatly.
For all filesystems mounted by you using mount(8), a line is added to /etc/mtab, where the contents of that line is related to the given mount command, but not to what the kernel did.
For the root filesystem, mount(8) writes an initial line in /etc/mtab taken from /etc/fstab. Again the information is from you, not from the kernel.
Quoted 4 lines
Quoted 2 lines
# dmesg | grep 'Kernel command' Kernel command line: ro root=/dev/hda2 rootfstype=ext2...
Quoted 1 line
/dev/root/ ext3 rw 0 0
It would be a bad bug if the kernel mounted its root filesystem with a type different from the type given in "rootfstype=". But I see you use an initrd, and there can be all kinds of commands there.
Thank you all for all your input. The tune2fs option was eventually used and we run into other problems. I think Andries was right in that the initrd was interfering, that's where we run into issues after the tune2fs.
I was trying to avoid the tune2fs as it involves booting into a live CD and brings the system down to where I can't access it over the network (it is a 4 hour drive).
At the end we had to replace the drive and recreate all file systems. If it ever happens again I will pay closer attention to the initrd commands to see if the rootfstype=ext2 was overridden with what's there.
Thanks,
Alberto
Quoted 26 lines · Andries Brouwer
Quoted 5 lines
Quoted 5 lines
You were right, even after making the changes, it seems to be telling lies:
# mount /dev/hda2 on / type ext2 (rw,usrquota)Roughly speaking:
/etc/mtabshows you what you said to mount./proc/mountsshows what the current kernel state is. These may differ greatly.For all filesystems mounted by you using mount(8), a line is added to
/etc/mtab, where the contents of that line is related to the given mount command, but not to what the kernel did.For the root filesystem, mount(8) writes an initial line in
/etc/mtabtaken from/etc/fstab. Again the information is from you, not from the kernel.Quoted 4 lines
Quoted 2 lines
# dmesg | grep 'Kernel command' Kernel command line: ro root=/dev/hda2 rootfstype=ext2...
Quoted 1 line
/dev/root/ ext3 rw 0 0It would be a bad bug if the kernel mounted its root filesystem with a type different from the type given in "rootfstype=". But I see you use an initrd, and there can be all kinds of commands there.
Signature · 3 lines
Alberto Alonso Global Gate Systems LLC. (512) 351-7233 http://www.ggsys.net Hardware, consulting, sysadmin, monitoring and remote backups