Skip to contentSkip to conversation

Ext3 mounted as ext2 but journal still in effect.

DISCUSSION Alberto Alonso 5 participants 7 messages Jan 12, 2007 – Jan 19, 2007

lore
Share & tools
lore
#0 lore raw

I have an ext3 filesystem that has been having problems with its journal. The result is that the file system remounts internally as read-only and the server becomes unusable, even shutdown does not work, using up 100% of the CPU but not rebooting.

I found some postings indicating that mounting it as ext2 should fix the problem, as it doesn't appear to be a hardware issue.

So, I decided to mount everything as ext2. Mount shows this:

# mount
/dev/hda2 on / type ext2 (rw,usrquota)
none on /proc type proc (rw)
none on /sys type sysfs (rw)
none on /dev/pts type devpts (rw,gid=5,mode=620)
usbfs on /proc/bus/usb type usbfs (rw)
/dev/hda1 on /boot type ext2 (rw)
none on /dev/shm type tmpfs (rw,noexec)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)

But now I still get the error:

# dmesg
[...]
EXT3-fs error (device hda2) in start_transaction: Journal has aborted
EXT3-fs error (device hda2) in start_transaction: Journal has aborted
EXT3-fs error (device hda2) in start_transaction: Journal has aborted
EXT3-fs error (device hda2) in start_transaction: Journal has aborted
[...]

The kernel is:

# uname -a
Linux hyperweb.net 2.6.5-1.358smp #1 SMP Sat May 8 09:25:36 EDT 2004
i686 i686 i386 GNU/Linux

Any ideas?

Thanks,

Alberto

Signature · 3 lines
Alberto Alonso                        Global Gate Systems LLC.
(512) 351-7233                        http://www.ggsys.net
Hardware, consulting, sysadmin, monitoring and remote backups
#1 lore raw
Quoted 34 lines · On Thu, 11 Jan 2007 23:08:16 -0600 Alberto Alonso <[email protected]> wrote:

I have an ext3 filesystem that has been having problems with its journal. The result is that the file system remounts internally as read-only and the server becomes unusable, even shutdown does not work, using up 100% of the CPU but not rebooting.

I found some postings indicating that mounting it as ext2 should fix the problem, as it doesn't appear to be a hardware issue.

So, I decided to mount everything as ext2. Mount shows this:

# mount
/dev/hda2 on / type ext2 (rw,usrquota)
none on /proc type proc (rw)
none on /sys type sysfs (rw)
none on /dev/pts type devpts (rw,gid=5,mode=620)
usbfs on /proc/bus/usb type usbfs (rw)
/dev/hda1 on /boot type ext2 (rw)
none on /dev/shm type tmpfs (rw,noexec)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)

But now I still get the error:

# dmesg
[...]
EXT3-fs error (device hda2) in start_transaction: Journal has aborted
EXT3-fs error (device hda2) in start_transaction: Journal has aborted
EXT3-fs error (device hda2) in start_transaction: Journal has aborted
EXT3-fs error (device hda2) in start_transaction: Journal has aborted
[...]
Quoted 9 lines

The kernel is:

# uname -a
Linux hyperweb.net 2.6.5-1.358smp #1 SMP Sat May 8 09:25:36 EDT 2004
i686 i686 i386 GNU/Linux

Any ideas?

mount(8) tells lies. Look in /proc/mounts and you'll see that it's really mounted as ext3.

You probably want to add `rootfstype=ext2' to the kernel boot command line.

#2 lore raw

You were right, even after making the changes, it seems to be telling lies:

# mount
/dev/hda2 on / type ext2 (rw,usrquota)
[...]

However, I think I am still not mounting as ext2:

# dmesg | grep 'Kernel command'
Kernel command line: ro root=/dev/hda2 rootfstype=ext2
# cat /proc/mounts
rootfs / rootfs rw 0 0
/dev/root / ext3 rw 0 0
/proc /proc proc rw,nodiratime 0 0
/sys /sys sysfs rw 0 0
none /dev/pts devpts rw 0 0
usbfs /proc/bus/usb usbfs rw 0 0
/dev/hda1 /boot ext2 rw 0 0
none /dev/shm tmpfs rw,noexec 0 0
none /proc/sys/fs/binfmt_misc binfmt_misc rw 0 0
sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw 0 0

Do I need to mess with the initrd? My grub lines look like this:

title Fedora Core (2.6.5-1.358smp)

        root (hd0,0)
        kernel /vmlinuz-2.6.5-1.358smp ro root=/dev/hda2 rootfstype=ext2
        initrd /initrd-2.6.5-1.358smp.img
title Fedora Core-up (2.6.5-1.358)
        root (hd0,0)
        kernel /vmlinuz-2.6.5-1.358 ro root=/dev/hda2 rootfstype=ext2
        initrd /initrd-2.6.5-1.358.img

Thanks,

Alberto

Quoted 52 lines · Andrew Morton
Quoted 34 lines · On Thu, 11 Jan 2007 23:08:16 -0600 Alberto Alonso <[email protected]> wrote:

I have an ext3 filesystem that has been having problems with its journal. The result is that the file system remounts internally as read-only and the server becomes unusable, even shutdown does not work, using up 100% of the CPU but not rebooting.

I found some postings indicating that mounting it as ext2 should fix the problem, as it doesn't appear to be a hardware issue.

So, I decided to mount everything as ext2. Mount shows this:

# mount
/dev/hda2 on / type ext2 (rw,usrquota)
none on /proc type proc (rw)
none on /sys type sysfs (rw)
none on /dev/pts type devpts (rw,gid=5,mode=620)
usbfs on /proc/bus/usb type usbfs (rw)
/dev/hda1 on /boot type ext2 (rw)
none on /dev/shm type tmpfs (rw,noexec)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)

But now I still get the error:

# dmesg
[...]
EXT3-fs error (device hda2) in start_transaction: Journal has aborted
EXT3-fs error (device hda2) in start_transaction: Journal has aborted
EXT3-fs error (device hda2) in start_transaction: Journal has aborted
EXT3-fs error (device hda2) in start_transaction: Journal has aborted
[...]
Quoted 9 lines

The kernel is:

# uname -a
Linux hyperweb.net 2.6.5-1.358smp #1 SMP Sat May 8 09:25:36 EDT 2004
i686 i686 i386 GNU/Linux

Any ideas?

mount(8) tells lies. Look in /proc/mounts and you'll see that it's really mounted as ext3.

You probably want to add `rootfstype=ext2' to the kernel boot command line.

Signature · 3 lines
Alberto Alonso                        Global Gate Systems LLC.
(512) 351-7233                        http://www.ggsys.net
Hardware, consulting, sysadmin, monitoring and remote backups
#3 lore raw

Hi!

Quoted 11 lines

You were right, even after making the changes, it seems to be telling lies:

# mount
/dev/hda2 on / type ext2 (rw,usrquota)
[...]

However, I think I am still not mounting as ext2:

# dmesg | grep 'Kernel command'
Kernel command line: ro root=/dev/hda2 rootfstype=ext2

...

Quoted 6 lines

rootfs / rootfs rw 0 0
/dev/root / ext3 rw 0 0

Do I need to mess with the initrd? My grub lines look like this:

Yes, probably.

Pavel

Signature · 1 line
Thanks for all the (sleeping) penguins.
#5 lore raw
In reply to Pavel Machek
Quoted 6 lines
Quoted 6 lines

You were right, even after making the changes, it seems to be telling lies:

# mount
/dev/hda2 on / type ext2 (rw,usrquota)

Roughly speaking:
/etc/mtab shows you what you said to mount. /proc/mounts shows what the current kernel state is. These may differ greatly.

For all filesystems mounted by you using mount(8), a line is added to /etc/mtab, where the contents of that line is related to the given mount command, but not to what the kernel did.

For the root filesystem, mount(8) writes an initial line in /etc/mtab taken from /etc/fstab. Again the information is from you, not from the kernel.

Quoted 4 lines
Quoted 2 lines
# dmesg | grep 'Kernel command'
Kernel command line: ro root=/dev/hda2 rootfstype=ext2

...

Quoted 1 line

/dev/root / ext3 rw 0 0

It would be a bad bug if the kernel mounted its root filesystem with a type different from the type given in "rootfstype=". But I see you use an initrd, and there can be all kinds of commands there.

#6 lore raw

Thank you all for all your input. The tune2fs option was eventually used and we run into other problems. I think Andries was right in that the initrd was interfering, that's where we run into issues after the tune2fs.

I was trying to avoid the tune2fs as it involves booting into a live CD and brings the system down to where I can't access it over the network (it is a 4 hour drive).

At the end we had to replace the drive and recreate all file systems. If it ever happens again I will pay closer attention to the initrd commands to see if the rootfstype=ext2 was overridden with what's there.

Thanks,

Alberto

Quoted 26 lines · Andries Brouwer
Quoted 5 lines
Quoted 5 lines

You were right, even after making the changes, it seems to be telling lies:

# mount
/dev/hda2 on / type ext2 (rw,usrquota)

Roughly speaking:
/etc/mtab shows you what you said to mount. /proc/mounts shows what the current kernel state is. These may differ greatly.

For all filesystems mounted by you using mount(8), a line is added to /etc/mtab, where the contents of that line is related to the given mount command, but not to what the kernel did.

For the root filesystem, mount(8) writes an initial line in /etc/mtab taken from /etc/fstab. Again the information is from you, not from the kernel.

Quoted 4 lines
Quoted 2 lines
# dmesg | grep 'Kernel command'
Kernel command line: ro root=/dev/hda2 rootfstype=ext2

...

Quoted 1 line

/dev/root / ext3 rw 0 0

It would be a bad bug if the kernel mounted its root filesystem with a type different from the type given in "rootfstype=". But I see you use an initrd, and there can be all kinds of commands there.

Signature · 3 lines
Alberto Alonso                        Global Gate Systems LLC.
(512) 351-7233                        http://www.ggsys.net
Hardware, consulting, sysadmin, monitoring and remote backups