mkfs.ext4 -D option fails to mount

Our validation team noticed that in some configurations mkfs.ext4 with the -D option creates a filesystem that can't be mounted:

  # mkfs.ext4 -D -F /dev/pmem5 
  mke2fs 1.43.3 (04-Sep-2016)
  /dev/pmem5 contains a ext4 file system
  	last mounted on Tue Jul 26 07:44:19 2016
  Creating filesystem with 65027584 4k blocks and 16261120 inodes
  Filesystem UUID: 6f95ece9-d4cb-4cfc-bc22-211119d5efe7
  Superblock backups stored on blocks: 
  	32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
  	4096000, 7962624, 11239424, 20480000, 23887872

Allocating group tables: done
Writing inode tables: done
Creating journal (262144 blocks): done
Writing superblocks and filesystem accounting information: done

  # mount /dev/pmem5 /mnt
  mount: wrong fs type, bad option, bad superblock on /dev/pmem5,
         missing codepage or helper program, or other error

In some cases useful info is found in syslog - try dmesg | tail or so.

where dmesg says:

EXT4-fs (pmem5): ext4_check_descriptors: Block bitmap for group 1 overlaps superblock EXT4-fs (pmem5): ext4_check_descriptors: Inode bitmap for group 1 overlaps superblock EXT4-fs (pmem5): ext4_check_descriptors: Inode table for group 1 overlaps superblock EXT4-fs (pmem5): ext4_check_descriptors: Block bitmap for group 2 overlaps superblock EXT4-fs (pmem5): ext4_check_descriptors: Inode bitmap for group 2 overlaps superblock EXT4-fs (pmem5): ext4_check_descriptors: Inode table for group 2 overlaps superblock
...
EXT4-fs (pmem5): ext4_check_descriptors: Block bitmap for group 63 overlaps superblock EXT4-fs (pmem5): ext4_check_descriptors: Inode bitmap for group 63 overlaps superblock EXT4-fs (pmem5): ext4_check_descriptors: Inode table for group 63 overlaps superblock EXT4-fs (pmem5): no journal found

If we omit the "-D" option from mkfs.ext4, everything works. Note also that this behavior is independent of the DAX mount option.

This isn't blocking us, I just thought you would want to know.

- Ross

One more bit of info - this seems to be strongly tied to the size of the block device. With a 32 GB block device it works fine, with 248 GB you get overlap messages for groups 1 through 63, and with a 250 GB device you get overlaps for groups 1 through 1999.

I've been varying my virtual NVDIMM namespace size via QEMU. Here's are the relevant bits from my QEMU my command line to enable the NVDIMM:

-m 8G,slots=3,maxmem=512G -machine pc,accel=kvm,nvdimm
-object memory-backend-file,id=mem1,share,mem-path=/tmp/nvdimm-8,size=250G -device nvdimm,memdev=mem1,id=nv1

Here's my full QEMU command line, in case that's interesting:

sudo /usr/bin/qemu-system-x86_64 /home/rzwisler/vms/amonkhet-8.qcow2 -m 8G,slots=3,maxmem=512G -smp 6 -machine pc,accel=kvm,nvdimm -enable-kvm -netdev tap,id=hostnet0,ifname=tap8,script=no,downscript=no -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:48:94:b8,bus=pci.0,addr=0x8 -rtc base=localtime -serial stdio -display none -monitor unix:/tmp/amonkhet-8.monitor,server,nowait -object memory-backend-file,id=mem1,share,mem-path=/tmp/nvdimm-8,size=250G -device nvdimm,memdev=mem1,id=nv1

- Ross

The -D option just means that we're doing the I/O using Direct I/O (as opposed to buffered I/O). It shouldn't make any difference to what gets written, so this very much smells like a bug in how /dev/pmem supports Direct I/O...

This very much sounds like Direct I/O is just getting completely botched for the pmem device, and writes to a block group descriptor block is affecting the wrong place on the storage device.

- Ted

This also reproduces with brd or loop as our block device:

  # modprobe brd rd_size=$((1024*1024*248))

  # mkfs.ext4 /dev/ram0 -F -D 
  mke2fs 1.43.3 (04-Sep-2016)
  Discarding device blocks: done                            
  Creating filesystem with 65011712 4k blocks and 16252928 inodes
  Filesystem UUID: 1632baa4-7260-4cc1-9558-0cb3aa2f213e
  Superblock backups stored on blocks: 
  	32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
  	4096000, 7962624, 11239424, 20480000, 23887872

Allocating group tables: done
Writing inode tables: done
Creating journal (262144 blocks): done
Writing superblocks and filesystem accounting information: done

  # mount /dev/ram0 /mnt
  mount: wrong fs type, bad option, bad superblock on /dev/ram0,
         missing codepage or helper program, or other error

In some cases useful info is found in syslog - try dmesg | tail or so.

where dmesg says:

EXT4-fs (ram0): ext4_check_descriptors: Block bitmap for group 64 overlaps superblock EXT4-fs (ram0): ext4_check_descriptors: Inode bitmap for group 64 overlaps superblock EXT4-fs (ram0): ext4_check_descriptors: Inode table for group 64 overlaps superblock EXT4-fs (ram0): ext4_check_descriptors: Block bitmap for group 65 overlaps superblock EXT4-fs (ram0): ext4_check_descriptors: Inode bitmap for group 65 overlaps superblock
...
EXT4-fs (ram0): ext4_check_descriptors: Block bitmap for group 1983 overlaps superblock EXT4-fs (ram0): ext4_check_descriptors: Inode bitmap for group 1983 overlaps superblock EXT4-fs (ram0): ext4_check_descriptors: Inode table for group 1983 overlaps superblock EXT4-fs (ram0): no journal found

or

  # truncate -s 248G loop_fs
  # losetup /dev/loop0 ./loop_fs 
  # mkfs.ext4 /dev/loop0 -F -D
  mke2fs 1.43.3 (04-Sep-2016)
  /dev/loop0 contains a ext4 file system
  	last mounted on Thu Jun 29 17:03:07 2017
  Discarding device blocks: done                            
  Creating filesystem with 65011712 4k blocks and 16252928 inodes
  Filesystem UUID: 00912ac9-599a-4396-9ef3-a353bdec69ea
  Superblock backups stored on blocks: 
  	32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
  	4096000, 7962624, 11239424, 20480000, 23887872

Allocating group tables: done
Writing inode tables: done
Creating journal (262144 blocks): done
Writing superblocks and filesystem accounting information: done

  # mount /dev/loop0 /tmp
  mount: wrong fs type, bad option, bad superblock on /dev/loop0,
         missing codepage or helper program, or other error

In some cases useful info is found in syslog - try dmesg | tail or so.

with similar messages.

- Ross

I'm not able to reproduce it using your reproduction receipe using e2fsprogs 1.43.4. I suspect you might be hitting a bug which was fixed by this commit:

commit d6cad379eb6c86ca58bf5b83a586577de412a2e6

Author: Theodore Ts'o <[email protected]>
Date:   Sun Sep 11 00:25:48 2016 -0400

libext2fs: fix unaligned, multiblock writes in the unix_io handler

The read-modify-write code for the unaligned fallback code wasn't working for multi-block writes. This was unmasked by FreeBSD 11-rc2, since its malloc() is returning unaligned memory regions for large memory regions.

Can you retry with e2fsprogs 1.43.4 and see if it fixes your issue?

- Ted

Yep, confirmed that it works with e2fsprogs v1.43.4. Cool, thanks.