Bug In ext4 in kernels > 2.6.39 - Not mounting with arguments/options I specify in fstab on root remount

DISCUSSION Matt Parnell 4 participants 18 messages Sep 13, 2011 – Nov 7, 2011 2 messages not in the archive ext4 file system

lore

Share & tools

lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

Matt Parnell

author

2011-09-13 09:07 UTC #1 lore raw

Hi all... I usually run my system the zen kernel (zen-kernel.org), but I've tried this with the default Archlinux kernel (pretty much a vanilla build) as well and for some reason, the root drive won't mount with the options I desire on anything above .29 The drive is an SSD, and this is really getting on my nerves. I've looked at the initscripts, fstab, everything else, and I just can't get any of this to work properly. It was mounting with all the options I specified in 2.6.39, but since some change to the kernel (or something else?) I can't mount with my options anymore. On top of this all, I've tried using both /dev/root and /dev/sdb1, but for some reason the kernel's only wanting to use /dev/root - I'm wondering if there's some kind of conflict due to that symlink...

I know this isn't my initscripts, as I've also tried this using openrc, aside from the default arch init system, and because upon downgrading to 2.6.39 everything is working normally again, I know it is the kernel. Information:

grub.cfg:

linux /boot/linux-zen root=/dev/sdb1 ro rootfstype=ext4
rootflags=data=writeback noatime barrier=0 discard noinitrd

fstab:

/dev/sdb1 / ext4 data=writeback,noatime,nodiratime,barrier=0,discard 0 1

Upon attempting to remount manually, I only get the following when I try to use the options I specified above:

EXT4-fs (sdb1): Cannot change data mode on remount

Interestingly, this also happened and worked, which I think is a bug, how can barrier=0 and 1 at the same time?

EXT4-fs (sdb1): re-mounted. Opts: user_xattr,barrier=1,barrier=0

I'm currently using a .39 kernel and can live with it for now, but I thought this was serious enough that the ext4 devs should hear about it. If you need any more information, just let me know.

Thanks!

Matt Parnell

Christian Kujau

2011-09-13 10:25 UTC #2 lore raw

Quoted 3 lines · Matt Parnell

I know this isn't my initscripts, as I've also tried this using openrc, aside from the default arch init system, and because upon downgrading to 2.6.39 everything is working normally again, I know it

Did you try booting with init=/bin/bash, i.e. bypassing all initscripts? Also, try booting w/o an initrd image if you can, thus bypassing all the early boot magic.

Shot in the dark: if it worked with 2.6.39 but not with 3.x and the initscript have not changed, maybe some initscript went mad when it could not parse the new kernel version number?

Christian.

Signature · 3 lines

BOFH excuse #141:

disks spinning backwards - toggle the hemisphere jumper.

A message that is not in the archive (sent off-list or not yet received): CABeR261eWW1vi1Xk9E6SWdy7PN=oiDbQdT_LiHgZf7+6V7Sv8Q@mail.gmail.com

Christian Kujau

2011-09-13 10:48 UTC #4 lore raw

[ Cc'ing linux-ext4 again ]

Quoted 8 lines · Matt Parnell

Quoted 3 lines
Did you try booting with init=/bin/bash, i.e. bypassing all initscripts? Also, try booting w/o an initrd image if you can, thus bypassing all the early boot magic.
Not exactly, but I had it drop out to perform an fsck, and the mtab indicated it was mounted according to my options, but this was when the disk was mounted read only from grub. Also, I don't use an initrd...so that's not it either.

Yeah, mtab is no good when your rootfs is readonly. What did /proc/mounts say?

Quoted 11 lines

Quoted 5 lines
Shot in the dark: if it worked with 2.6.39 but not with 3.x and the initscript have not changed, maybe some initscript went mad when it could not parse the new kernel version number?
Christian.
It's not that either... Arch has been using 3.0+ as it's default for a few weeks now, so they would've updated their scripts for that. Furthermore, I tried using OpenRC (the gentoo init system) and got the same results.

Signature · 3 lines

BOFH excuse #359:

YOU HAVE AN I/O ERROR -> Incompetent Operator error

Eric Sandeen

2011-09-13 14:49 UTC #5 lore raw

In reply to Matt Parnell

Quoted 4 lines · Matt Parnell

Interestingly, this also happened and worked, which I think is a bug, how can barrier=0 and 1 at the same time?
EXT4-fs (sdb1): re-mounted. Opts: user_xattr,barrier=1,barrier=0

Fairly certain that the parser will just take the last value specified for the option.

/proc/mounts likely says "nobarrier" when you specify it this way?

-Eric

Matt Parnell

author

2011-10-09 22:10 UTC #6 lore raw

FYI this behavior still occurs with every revision up to the current 3.0.6 working tree. I do not think it's my distro's initscripts, they've been rolling 3.0.X kernels for some time and those behave the same way, so it's not some sort of parsing issues - Arch's initscripts are all bash based for the most part anyway, see the attached from my system - that's the entire boot initscript minus my daemons.

There's no alternative init system for Archlinux at this point, I'm working on porting OpenRC from gentoo as I have time to write initscripts from scratch, but that's a slow moving project.

Whatever the case, the following are from my most recent attempt with a 3.0.6 build. Manually attempting to remount rw still fails, and no matter whether I use /dev/sdb1 or /dev/root, I get the same behavior.

/proc/mounts (as you can see it's not being remounted rw for some reason - /dev/sdc1 is the flash drive I used to pipe this info onto):

rootfs / rootfs rw 0 0
/dev/root / ext4 ro,relatime,barrier=1 0 0
devtmpfs /dev devtmpfs rw,relatime,size=4059824k,nr_inodes=1014956,mode=755 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
/sys /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
/run /run tmpfs rw,nosuid,nodev,relatime,size=10240k,mode=755 0 0
devpts /dev/pts devpts rw,relatime,mode=600 0 0
shm /dev/shm tmpfs rw,nosuid,nodev,relatime 0 0
fusectl /sys/fs/fuse/connections fusectl rw,relatime 0 0
/dev/sda4 /media/disk ext3 rw,nosuid,nodev,noatime,nodiratime,errors=continue,barrier=0,data=writeback
0 0
/dev/sdc1 /mnt/tmp vfat rw,relatime,fmask=0022,dmask=0022,codepage=cp437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 0 0

/etc/mtab (not useful, I know, but for reference):

/dev/root / ext4 rw,noatime,nodiratime,data=writeback,barrier=0,discard,commit=0 0 0 devtmpfs /dev devtmpfs rw,relatime,size=4059656k,nr_inodes=1014914,mode=755 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
/sys /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
/run /run tmpfs rw,nosuid,nodev,relatime,size=10240k,mode=755 0 0 devpts /dev/pts devpts rw,relatime,mode=600 0 0
shm /dev/shm tmpfs rw,nosuid,nodev,relatime 0 0
tmpfs /tmp tmpfs rw,nosuid,nodev 0 0

/etc/fstab:

#
# /etc/fstab: static file system information
#

devpts                 /dev/pts      devpts    defaults            0      0
shm                    /dev/shm      tmpfs     nodev,nosuid        0      0

/dev/sdb1 / ext4 defaults,noatime,nodiratime,data=writeback,barrier=0,discard 0 0 /dev/sda4 /media/disk ext3 defaults,data=writeback,noatime,nodiratime,user,exec 0 0
tmpfs /tmp tmpfs nodev,nosuid 0 0

Quoted 15 lines · Eric Sandeen

On 9/13/11 4:01 AM, Matt Parnell wrote:
Interestingly, this also happened and worked, which I think is a bug, how can barrier=0 and 1 at the same time?
EXT4-fs (sdb1): re-mounted. Opts: user_xattr,barrier=1,barrier=0
Fairly certain that the parser will just take the last value specified for the option.
/proc/mounts likely says "nobarrier" when you specify it this way?
-Eric

Ted Ts'o

maintainer

2011-10-09 23:44 UTC #7 lore raw

Quoted 6 lines · Matt Parnell

/proc/mounts (as you can see it's not being remounted rw for some reason - /dev/sdc1 is the flash drive I used to pipe this info onto):
rootfs / rootfs rw 0 0
/dev/root / ext4 ro,relatime,barrier=1 0 0

Can you send me the output of /proc/cmdline? It looks like the rootflags command line option isn't getting passed to the kernel for some reason. That would explain why data=writeback isn't showing up in /proc/mounts.

- Ted

Matt Parnell

author

2011-10-21 03:29 UTC #8 lore raw

The output of /proc/cmdline on 3.0.7:

BOOT_IMAGE=/boot/vmlinuz26-zen root=/dev/sdb1 ro rootfstype=ext4 rootflags=data=writeback noatime nodiratime barrier=0 discard noinitrd

Quoted 12 lines · Ted Ts'o

Quoted 5 lines · Matt Parnell
/proc/mounts (as you can see it's not being remounted rw for some reason - /dev/sdc1 is the flash drive I used to pipe this info onto):
rootfs / rootfs rw 0 0
/dev/root / ext4 ro,relatime,barrier=1 0 0
Can you send me the output of /proc/cmdline? It looks like the rootflags command line option isn't getting passed to the kernel for some reason. That would explain why data=writeback isn't showing up in /proc/mounts.
- Ted

Matt Parnell

author

2011-10-21 04:29 UTC #9 lore raw

Also, note that the image is named vmlinuz26-zen there by mistake - I fixed that in my build script, but even so, it makes no difference. In 3.0.7, I still can't remount root rw on ext4.

Quoted 18 lines · Matt Parnell

The output of /proc/cmdline on 3.0.7:
BOOT_IMAGE=/boot/vmlinuz26-zen root=/dev/sdb1 ro rootfstype=ext4 rootflags=data=writeback noatime nodiratime barrier=0 discard noinitrd
Quoted 12 lines · Ted Ts'o
Quoted 5 lines · Matt Parnell
/proc/mounts (as you can see it's not being remounted rw for some reason - /dev/sdc1 is the flash drive I used to pipe this info onto):
rootfs / rootfs rw 0 0
/dev/root / ext4 ro,relatime,barrier=1 0 0
Can you send me the output of /proc/cmdline? It looks like the rootflags command line option isn't getting passed to the kernel for some reason. That would explain why data=writeback isn't showing up in /proc/mounts.
- Ted

Ted Ts'o

maintainer

2011-10-22 07:49 UTC #10 lore raw

In reply to Matt Parnell

Quoted 14 lines · Matt Parnell

The output of /proc/cmdline on 3.0.7:
BOOT_IMAGE=/boot/vmlinuz26-zen root=/dev/sdb1 ro rootfstype=ext4 rootflags=data=writeback noatime nodiratime barrier=0 discard noinitrd
Quoted 6 lines · Ted Ts'o
Quoted 5 lines · Matt Parnell
/proc/mounts (as you can see it's not being remounted rw for some reason - /dev/sdc1 is the flash drive I used to pipe this info onto):
rootfs / rootfs rw 0 0
/dev/root / ext4 ro,relatime,barrier=1 0 0

I haven't had a chance to check on v3.0 (I'm currently travelling and penning this from an airplane), but on v3.1-rc3, it works for me under KVM. I fire up kvm-qemu this way:

$QEMU -enable-kvm -boot order=c $NET \
      -drive file=$ROOT_QCOW2,if=virtio$SNAPSHOT \
      -drive file=$VDB,cache=none,if=virtio \
      -drive file=$VDC,cache=none,if=virtio \
      -drive file=$VDD,cache=none,if=virtio \
      -nographic -smp $NR_CPU -m $MEM \
      --kernel $KERNEL \
      --append "root=/dev/vda console=ttyS0,115200 rootflags=data=writeback" |\

tee $LOGFILE

with this in /proc/cmdline:

candygram:~# cat /proc/cmdline 
root=/dev/vda console=ttyS0,115200 rootflags=data=writeback maint

... and this is what I have in /proc/mounts:

candygram:~# head -2 /proc/mounts
rootfs / rootfs rw 0 0
/dev/root / ext4 rw,noatime,user_xattr,acl,barrier=1,data=writeback 0 0

... and what I have in /etc/fstab:

UUID=ce4d6b98-2aa9-42d4-a127-995a795a0b02 / ext4 noatime,data=writeback 0 1

The kvm image I'm using is using Debian unstable image generated using debootstrap, and uses no modules or initrd's --- which makes it easier for me to boot since $KERNEL is set to:

    /tyt/linux/ext4/arch/x86/boot/bzImage

This allows me to fire up test kernel right from my build tree, without needing to set up any kind of initrd nonsense. :-)

- Ted

Matt Parnell

author

2011-10-22 07:51 UTC #11 lore raw

That doesn't really help me at all, it's not rootflags=data=writeback causing this, it's starting to make me think that arch's init may be to blame, although I previously ruled it out...

This is all a big wtf though. I'm going to go complain at the arch devs now... I've not finished porting openrc to it yet.

Quoted 54 lines · Ted Ts'o

Quoted 14 lines · Matt Parnell
The output of /proc/cmdline on 3.0.7:
BOOT_IMAGE=/boot/vmlinuz26-zen root=/dev/sdb1 ro rootfstype=ext4 rootflags=data=writeback noatime nodiratime barrier=0 discard noinitrd
Quoted 6 lines · Ted Ts'o
Quoted 5 lines · Matt Parnell
/proc/mounts (as you can see it's not being remounted rw for some reason - /dev/sdc1 is the flash drive I used to pipe this info onto):
rootfs / rootfs rw 0 0
/dev/root / ext4 ro,relatime,barrier=1 0 0
I haven't had a chance to check on v3.0 (I'm currently travelling and penning this from an airplane), but on v3.1-rc3, it works for me under KVM. I fire up kvm-qemu this way:
$QEMU -enable-kvm -boot order=c $NET \
       -drive file=$ROOT_QCOW2,if=virtio$SNAPSHOT \
       -drive file=$VDB,cache=none,if=virtio \
       -drive file=$VDC,cache=none,if=virtio \
       -drive file=$VDD,cache=none,if=virtio \
       -nographic -smp $NR_CPU -m $MEM \
       --kernel $KERNEL \
       --append "root=/dev/vda console=ttyS0,115200 rootflags=data=writeback" |\
tee $LOGFILE
with this in /proc/cmdline:
candygram:~# cat /proc/cmdline
root=/dev/vda console=ttyS0,115200 rootflags=data=writeback maint
... and this is what I have in /proc/mounts:
candygram:~# head -2 /proc/mounts
rootfs / rootfs rw 0 0
/dev/root / ext4 rw,noatime,user_xattr,acl,barrier=1,data=writeback 0 0
... and what I have in /etc/fstab:
UUID=ce4d6b98-2aa9-42d4-a127-995a795a0b02 / ext4 noatime,data=writeback 0 1
The kvm image I'm using is using Debian unstable image generated using debootstrap, and uses no modules or initrd's --- which makes it easier for me to boot since $KERNEL is set to:
     /tyt/linux/ext4/arch/x86/boot/bzImage
This allows me to fire up test kernel right from my build tree, without needing to set up any kind of initrd nonsense. :-)
- Ted

Matt Parnell

author

2011-10-22 07:53 UTC #12 lore raw

Also, I didn't intend to come across cross there... I appreciate the efforts you are taking there.

It's just, I don't know what could've changed between .29 and 3.0 that would break an entire init system so that it won't remount root rw...

Quoted 60 lines · Matt Parnell

That doesn't really help me at all, it's not rootflags=data=writeback causing this, it's starting to make me think that arch's init may be to blame, although I previously ruled it out...
This is all a big wtf though. I'm going to go complain at the arch devs now... I've not finished porting openrc to it yet.
Quoted 55 lines · Ted Ts'o
Quoted 14 lines · Matt Parnell
The output of /proc/cmdline on 3.0.7:
BOOT_IMAGE=/boot/vmlinuz26-zen root=/dev/sdb1 ro rootfstype=ext4 rootflags=data=writeback noatime nodiratime barrier=0 discard noinitrd
Quoted 6 lines · Ted Ts'o
Quoted 5 lines · Matt Parnell
/proc/mounts (as you can see it's not being remounted rw for some reason - /dev/sdc1 is the flash drive I used to pipe this info onto):
rootfs / rootfs rw 0 0
/dev/root / ext4 ro,relatime,barrier=1 0 0
I haven't had a chance to check on v3.0 (I'm currently travelling and penning this from an airplane), but on v3.1-rc3, it works for me under KVM. I fire up kvm-qemu this way:
$QEMU -enable-kvm -boot order=c $NET \
       -drive file=$ROOT_QCOW2,if=virtio$SNAPSHOT \
       -drive file=$VDB,cache=none,if=virtio \
       -drive file=$VDC,cache=none,if=virtio \
       -drive file=$VDD,cache=none,if=virtio \
       -nographic -smp $NR_CPU -m $MEM \
       --kernel $KERNEL \
       --append "root=/dev/vda console=ttyS0,115200 
rootflags=data=writeback" |\
tee $LOGFILE
with this in /proc/cmdline:
candygram:~# cat /proc/cmdline
root=/dev/vda console=ttyS0,115200 rootflags=data=writeback maint
... and this is what I have in /proc/mounts:
candygram:~# head -2 /proc/mounts
rootfs / rootfs rw 0 0
/dev/root / ext4 rw,noatime,user_xattr,acl,barrier=1,data=writeback 0 0
... and what I have in /etc/fstab:
UUID=ce4d6b98-2aa9-42d4-a127-995a795a0b02 / ext4 noatime,data=writeback 0 1
The kvm image I'm using is using Debian unstable image generated using debootstrap, and uses no modules or initrd's --- which makes it easier for me to boot since $KERNEL is set to:
     /tyt/linux/ext4/arch/x86/boot/bzImage
This allows me to fire up test kernel right from my build tree, without needing to set up any kind of initrd nonsense. :-)
- Ted

Ted Ts'o

maintainer

2011-10-22 09:32 UTC #13 lore raw

In reply to Matt Parnell

Quoted 4 lines · Matt Parnell

That doesn't really help me at all, it's not rootflags=data=writeback causing this, it's starting to make me think that arch's init may be to blame, although I previously ruled it out...

Well, it looks like rootflags=data=writeback is not making it to the file system. That's why it's not showing up in /proc/mounts, from you showed us. Can you look at the kernel dmesg?

You should see something like this:

[    1.421146] EXT3-fs (vda): error: couldn't mount because of unsupported optional features (240)
[    1.434057] EXT4-fs (vda): couldn't mount as ext2 due to feature incompatibilities
[    1.454631] EXT4-fs (vda): mounted filesystem with writeback data mode. Opts: data=writeback
[    1.455966] VFS: Mounted root (ext4 filesystem) readonly on device 254:0.

The first line is the failure to mount the root file system as ext3. The 2nd is the failure to mount the file system as ext2, using the ext4 file system driver. The last two lines show the options show the mount as ext4.

What do those two lines look to you. If you don't see "Opts:
data=writeback", then somehow the rootflags option isn't getting passed down to the file system. Then when you try to remount the file system read/write, the fact that you have "data=writeback" in your /etc/fstab causes the failure to remount.

If you simply remove that from /etc/fstab, things should work better. The remount will preserve whatever data=journalling mode was in use when the root file system was originallymounted as. If rootflags is non-functional, then the file system won't be mounted as data=writeback, but at least the boot sequence will continue without blowing out.

- Ted

Matt Parnell

author

2011-11-07 02:18 UTC #14 lore raw

Quoted 28 lines · Ted Ts'o

Quoted 4 lines · Matt Parnell
That doesn't really help me at all, it's not rootflags=data=writeback causing this, it's starting to make me think that arch's init may be to blame, although I previously ruled it out...
Well, it looks like rootflags=data=writeback is not making it to the file system. That's why it's not showing up in /proc/mounts, from you showed us. Can you look at the kernel dmesg?
You should see something like this:
[    1.421146] EXT3-fs (vda): error: couldn't mount because of unsupported optional features (240)
[    1.434057] EXT4-fs (vda): couldn't mount as ext2 due to feature incompatibilities
[    1.454631] EXT4-fs (vda): mounted filesystem with writeback data mode. Opts: data=writeback
[    1.455966] VFS: Mounted root (ext4 filesystem) readonly on device 254:0.
The first line is the failure to mount the root file system as ext3. The 2nd is the failure to mount the file system as ext2, using the ext4 file system driver. The last two lines show the options show the mount as ext4.
What do those two lines look to you. If you don't see "Opts:
data=writeback", then somehow the rootflags option isn't getting passed down to the file system. Then when you try to remount the file system read/write, the fact that you have "data=writeback" in your /etc/fstab causes the failure to remount.

Quoted 8 lines

If you simply remove that from /etc/fstab, things should work better. The remount will preserve whatever data=journalling mode was in use when the root file system was originallymounted as. If rootflags is non-functional, then the file system won't be mounted as data=writeback, but at least the boot sequence will continue without blowing out.
- Ted

Since then, I created a fresh partition and re rsynced my root over onto it after making it WITHOUT a journal using the current sysrescue64 cd (kernel 3.0.1).

Same thing is happening, and the drive still refuses to mount any other way except

/dev/root / ext4 rw,relatime,user_xattr,acl,barrier=1 0 0

Despite my specifying these in fstab and grub.cfg ...it's like it's not even passing the options somewhere, either from grub2 to the kernel or from my initscripts...but since I've tried using openrc too and get this result, I'm not sure what to do. Note that using both /dev/sdb1 and /dev/root don't interchange between the fstab and the kernel...I think the /dev/root symlink may be breaking my initscripts.

/dev/sdb1 / ext4 noatime,nodiratime,data=writeback,nouser_xattr,barrier=0,commit=10 0 0

Matt Parnell

author

2011-11-07 02:24 UTC #15 lore raw

In reply to Ted Ts'o

Quoted 28 lines · Ted Ts'o

Quoted 4 lines · Matt Parnell
That doesn't really help me at all, it's not rootflags=data=writeback causing this, it's starting to make me think that arch's init may be to blame, although I previously ruled it out...
Well, it looks like rootflags=data=writeback is not making it to the file system. That's why it's not showing up in /proc/mounts, from you showed us. Can you look at the kernel dmesg?
You should see something like this:
[    1.421146] EXT3-fs (vda): error: couldn't mount because of unsupported optional features (240)
[    1.434057] EXT4-fs (vda): couldn't mount as ext2 due to feature incompatibilities
[    1.454631] EXT4-fs (vda): mounted filesystem with writeback data mode. Opts: data=writeback
[    1.455966] VFS: Mounted root (ext4 filesystem) readonly on device 254:0.
The first line is the failure to mount the root file system as ext3. The 2nd is the failure to mount the file system as ext2, using the ext4 file system driver. The last two lines show the options show the mount as ext4.
What do those two lines look to you. If you don't see "Opts:
data=writeback", then somehow the rootflags option isn't getting passed down to the file system. Then when you try to remount the file system read/write, the fact that you have "data=writeback" in your /etc/fstab causes the failure to remount.

Quoted 8 lines

If you simply remove that from /etc/fstab, things should work better. The remount will preserve whatever data=journalling mode was in use when the root file system was originallymounted as. If rootflags is non-functional, then the file system won't be mounted as data=writeback, but at least the boot sequence will continue without blowing out.
- Ted

Looks like the kernel isn't creating the /dev/root link/block device, either all of the time or some of the time, I'm confused.

Matt Parnell

author

2011-11-07 03:27 UTC #16 lore raw

Quoted 36 lines · Matt Parnell

Quoted 35 lines · Ted Ts'o
Quoted 4 lines · Matt Parnell
That doesn't really help me at all, it's not rootflags=data=writeback causing this, it's starting to make me think that arch's init may be to blame, although I previously ruled it out...
Well, it looks like rootflags=data=writeback is not making it to the file system. That's why it's not showing up in /proc/mounts, from you showed us. Can you look at the kernel dmesg?
You should see something like this:
[    1.421146] EXT3-fs (vda): error: couldn't mount because of unsupported optional features (240)
[    1.434057] EXT4-fs (vda): couldn't mount as ext2 due to feature incompatibilities
[    1.454631] EXT4-fs (vda): mounted filesystem with writeback data mode. Opts: data=writeback
[    1.455966] VFS: Mounted root (ext4 filesystem) readonly on device 254:0.
The first line is the failure to mount the root file system as ext3. The 2nd is the failure to mount the file system as ext2, using the ext4 file system driver. The last two lines show the options show the mount as ext4.
What do those two lines look to you. If you don't see "Opts:
data=writeback", then somehow the rootflags option isn't getting passed down to the file system. Then when you try to remount the file system read/write, the fact that you have "data=writeback" in your /etc/fstab causes the failure to remount.
If you simply remove that from /etc/fstab, things should work better. The remount will preserve whatever data=journalling mode was in use when the root file system was originallymounted as. If rootflags is non-functional, then the file system won't be mounted as data=writeback, but at least the boot sequence will continue without blowing out.

Quoted 2 lines

Quoted 1 line
- Ted
Looks like the kernel isn't creating the /dev/root link/block device, either all of the time or some of the time, I'm confused.

I figured it out. data=writeback isn't needed and is done by default if there's no journal, as per commit 373cd5c53d5ea6622c319ecd84e29e2737d488bd http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=373cd5c53d5ea6622c319ecd84e29e2737d488bd

...sorry about throwing you guys through loops over this, although I'm guessing there's still a bug in that the kernel should just warn the user about the option when it's used and otherwise ignore it instead of breaking if there's no journal.

Matt Parnell

author

2011-11-07 03:33 UTC #17 lore raw

Quoted 37 lines · Matt Parnell

Quoted 36 lines · Matt Parnell
Quoted 35 lines · Ted Ts'o
Quoted 4 lines · Matt Parnell
That doesn't really help me at all, it's not rootflags=data=writeback causing this, it's starting to make me think that arch's init may be to blame, although I previously ruled it out...
Well, it looks like rootflags=data=writeback is not making it to the file system. That's why it's not showing up in /proc/mounts, from you showed us. Can you look at the kernel dmesg?
You should see something like this:
[    1.421146] EXT3-fs (vda): error: couldn't mount because of unsupported optional features (240)
[    1.434057] EXT4-fs (vda): couldn't mount as ext2 due to feature incompatibilities
[    1.454631] EXT4-fs (vda): mounted filesystem with writeback data mode. Opts: data=writeback
[    1.455966] VFS: Mounted root (ext4 filesystem) readonly on device 254:0.
The first line is the failure to mount the root file system as ext3. The 2nd is the failure to mount the file system as ext2, using the ext4 file system driver. The last two lines show the options show the mount as ext4.
What do those two lines look to you. If you don't see "Opts:
data=writeback", then somehow the rootflags option isn't getting passed down to the file system. Then when you try to remount the file system read/write, the fact that you have "data=writeback" in your /etc/fstab causes the failure to remount.
If you simply remove that from /etc/fstab, things should work better. The remount will preserve whatever data=journalling mode was in use when the root file system was originallymounted as. If rootflags is non-functional, then the file system won't be mounted as data=writeback, but at least the boot sequence will continue without blowing out.

Quoted 5 lines

Quoted 2 lines
Quoted 1 line
- Ted
Looks like the kernel isn't creating the /dev/root link/block device, either all of the time or some of the time, I'm confused.
I figured it out. data=writeback isn't needed and is done by default if there's no journal, as per commit 373cd5c53d5ea6622c319ecd84e29e2737d488bd http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=373cd5c53d5ea6622c319ecd84e29e2737d488bd
...sorry about throwing you guys through loops over this, although I'm guessing there's still a bug in that the kernel should just warn the user about the option when it's used and otherwise ignore it instead of breaking if there's no journal.

That is, you were right in removing rootflags=data=writeback from the kernel boot params, and removing data=writeback from fstab would fix it - that commit is where things break, though...

So really instead of breaking it should log that it's ignoring that flag becuase it's enabled by default when there's no journal.

Ted Ts'o

maintainer

2011-11-07 15:48 UTC #18 lore raw

In reply to Matt Parnell

Quoted 8 lines · Matt Parnell

I figured it out. data=writeback isn't needed and is done by default
if there's no journal, as per commit
373cd5c53d5ea6622c319ecd84e29e2737d488bd http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=373cd5c53d5ea6622c319ecd84e29e2737d488bd
...sorry about throwing you guys through loops over this, although I'm guessing there's still a bug in that the kernel should just warn the user about the option when it's used and otherwise ignore it instead of breaking if there's no journal.

Ah, OK. It wasn't clear to me that you were running without a year. You're right, we shouldn't complain in that case, since we ignore the messages when mounting the file system. Actually, we parse the options when we are mounting the file system, but then we zero them out when we discover that there's no journal; but in any case, we should similarly ignore the pointless data=* mount options (for a file system w/o a journal) on the remount operation as well.

- Ted

commit eb513689c97e3e73bb9b4459d490a8e894b4a546

Author: Theodore Ts'o <[email protected]>
Date:   Mon Nov 7 10:47:42 2011 -0500

ext4: ignore journalled data options on remount if fs has no journal

This avoids a confusing failure in the init scripts when the /etc/fstab has data=writeback or data=journal but the file system does not have a journal. So check for this case explicitly, and warn the user that we are ignoring the (pointless, since they have no journal) data=* mount option.

Signed-off-by "Theodore Ts'o" <[email protected]>

fs/ext4/super.c +3 -1

index 9953d80..0435013 100644
@@ -1683,7 +1683,9 @@ static int parse_options(char *options, struct super_block *sb,
 			data_opt = EXT4_MOUNT_WRITEBACK_DATA;
 		datacheck:
 			if (is_remount) {
-				if (test_opt(sb, DATA_FLAGS) != data_opt) {
+				if (!sbi->s_journal)
+					ext4_msg(sb, KERN_WARNING, "Remounting file system with no journal so ignoring journalled data option");
+				else if (test_opt(sb, DATA_FLAGS) != data_opt) {
 					ext4_msg(sb, KERN_ERR,
 						"Cannot change data mode on remount");
 					return 0;

Matt Parnell

author

2011-11-07 23:49 UTC #19 lore raw

Quoted 38 lines · Ted Ts'o

Quoted 8 lines · Matt Parnell
I figured it out. data=writeback isn't needed and is done by default
if there's no journal, as per commit
373cd5c53d5ea6622c319ecd84e29e2737d488bd http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=373cd5c53d5ea6622c319ecd84e29e2737d488bd
...sorry about throwing you guys through loops over this, although I'm guessing there's still a bug in that the kernel should just warn the user about the option when it's used and otherwise ignore it instead of breaking if there's no journal.
Ah, OK. It wasn't clear to me that you were running without a year. You're right, we shouldn't complain in that case, since we ignore the messages when mounting the file system. Actually, we parse the options when we are mounting the file system, but then we zero them out when we discover that there's no journal; but in any case, we should similarly ignore the pointless data=* mount options (for a file system w/o a journal) on the remount operation as well.
- Ted
commit eb513689c97e3e73bb9b4459d490a8e894b4a546
Author: Theodore Ts'o<[email protected]>
Date:   Mon Nov 7 10:47:42 2011 -0500
ext4: ignore journalled data options on remount if fs has no journal
This avoids a confusing failure in the init scripts when the /etc/fstab has data=writeback or data=journal but the file system does not have a journal. So check for this case explicitly, and warn the user that we are ignoring the (pointless, since they have no journal) data=* mount option.
Signed-off-by "Theodore Ts'o"<[email protected]>
fs/ext4/super.c +0 -0
index 9953d80..0435013 100644

fs/ext4/super.c static int parse_options(char *options, struct super_block *sb, · line 1691

Quoted 10 lines

fs/ext4/super.c +3 -1

@@ -1683,7 +1683,9 @@ static int parse_options(char *options, struct super_block *sb,
  			data_opt = EXT4_MOUNT_WRITEBACK_DATA;
  		datacheck:
  			if (is_remount) {
-				if (test_opt(sb, DATA_FLAGS) != data_opt) {
+				if (!sbi->s_journal)
+					ext4_msg(sb, KERN_WARNING, "Remounting file system with no journal so ignoring journalled data option");
+				else if (test_opt(sb, DATA_FLAGS) != data_opt) {
  					ext4_msg(sb, KERN_ERR,
  						"Cannot change data mode on remount");
  					return 0;

Great! I'm glad we managed to track this one down. I'm not using the flags anymore, but I'll be glad when your patch there makes it into ext4 for other users.