Skip to contentSkip to conversation

NFSv4 O_DIRECT with 3.16.2 kernel issue (qemu kvm cache=none / RHEL5 guest)

DISCUSSION Zdenek Kaspar 2 participants 4 messages Sep 12, 2014 – Oct 6, 2014

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hello everyone, got this specific problem with RHEL5 guest under KVM hypervisor.

client/server kernel version: 3.16.2

/etc/exports: rw,async,no_subtree_check,no_root_squash /proc/mounts: nfs4 rw,relatime,vers=4.0,rsize=524288,wsize=524288,

              namlen=255,hard,proto=tcp,port=0,
              timeo=600,retrans=2,sec=sys,clientaddr=xxx.xxx.xxx.xxx,
              local_lock=none,addr=xxx.xxx.xxx.yyy 0 0

CONFIG_NFS_FS=m
# CONFIG_NFS_V2 is not set
CONFIG_NFS_V3=m
# CONFIG_NFS_V3_ACL is not set
CONFIG_NFS_V4=m
# CONFIG_NFS_SWAP is not set
# CONFIG_NFS_V4_1 is not set
# CONFIG_NFS_USE_LEGACY_DNS is not set
CONFIG_NFS_USE_KERNEL_DNS=y
CONFIG_NFSD=m
CONFIG_NFSD_V3=y
# CONFIG_NFSD_V3_ACL is not set
CONFIG_NFSD_V4=y
# CONFIG_NFSD_FAULT_INJECTION is not set
CONFIG_NFS_COMMON=y

CentOS-5.10-i386-bin-1of8.iso -- did not test everything on centos, behaved same as RHEL5. So let's use it as VM reproducer:

dd if=/dev/zero of=kvm-test1.img bs=1M count=2048

start VM using:
file=kvm-test1.img,cache=writeback,aio=native,if=virtio,boot=on

create custom layout: /dev/vda1 / ext3 max size primary unselect Desktop-Gnome, select Customize now use only: Base System/Base and install it..

[root@test1 ~]# find / >/dev/null
everything works as expected with writeback

shutdown, backup kvm-test1.img (fs will explode) start VM using:
file=kvm-test1.img,cache=none,aio=native,if=virtio,boot=on

[root@test1 ~]# find / >/dev/null
and check dmesg, reboot etc, ie:
EXT3-fs error (device vda1): htree_dirblock_to_tree: bad entry in directory #460317: rec_len is smaller than minimal - offset=0, inode=0, rec_len=0, name_len=0
EXT3-fs error (device vda1): htree_dirblock_to_tree: bad entry in directory #98503: rec_len is smaller than minimal - offset=0, inode=0, rec_len=0, name_len=0

Tested locally on ext4, xfs - WORK
Tested on NFSv4 share with ext4, xfs - FAIL

Tested NFSv4 share sync, async       - FAIL
Tested on NFSv3 share with xfs       - WORK
KVM with qemu rhel-6.5, 2.1.1        - SAME RESULTS

NOTE: it seems just the rhel5 guest triggers it easily..

HTH, Z.

#1 lore raw
Quoted 61 lines · Zdenek Kaspar

Hello everyone, got this specific problem with RHEL5 guest under KVM hypervisor.

client/server kernel version: 3.16.2

/etc/exports: rw,async,no_subtree_check,no_root_squash /proc/mounts: nfs4 rw,relatime,vers=4.0,rsize=524288,wsize=524288,

              namlen=255,hard,proto=tcp,port=0,
              timeo=600,retrans=2,sec=sys,clientaddr=xxx.xxx.xxx.xxx,
              local_lock=none,addr=xxx.xxx.xxx.yyy 0 0

CONFIG_NFS_FS=m
# CONFIG_NFS_V2 is not set
CONFIG_NFS_V3=m
# CONFIG_NFS_V3_ACL is not set
CONFIG_NFS_V4=m
# CONFIG_NFS_SWAP is not set
# CONFIG_NFS_V4_1 is not set
# CONFIG_NFS_USE_LEGACY_DNS is not set
CONFIG_NFS_USE_KERNEL_DNS=y
CONFIG_NFSD=m
CONFIG_NFSD_V3=y
# CONFIG_NFSD_V3_ACL is not set
CONFIG_NFSD_V4=y
# CONFIG_NFSD_FAULT_INJECTION is not set
CONFIG_NFS_COMMON=y

CentOS-5.10-i386-bin-1of8.iso -- did not test everything on centos, behaved same as RHEL5. So let's use it as VM reproducer:

dd if=/dev/zero of=kvm-test1.img bs=1M count=2048

start VM using:
file=kvm-test1.img,cache=writeback,aio=native,if=virtio,boot=on

create custom layout: /dev/vda1 / ext3 max size primary unselect Desktop-Gnome, select Customize now use only: Base System/Base and install it..

[root@test1 ~]# find / >/dev/null
everything works as expected with writeback

shutdown, backup kvm-test1.img (fs will explode) start VM using:
file=kvm-test1.img,cache=none,aio=native,if=virtio,boot=on

[root@test1 ~]# find / >/dev/null
and check dmesg, reboot etc, ie:
EXT3-fs error (device vda1): htree_dirblock_to_tree: bad entry in directory #460317: rec_len is smaller than minimal - offset=0, inode=0, rec_len=0, name_len=0
EXT3-fs error (device vda1): htree_dirblock_to_tree: bad entry in directory #98503: rec_len is smaller than minimal - offset=0, inode=0, rec_len=0, name_len=0

Tested locally on ext4, xfs - WORK
Tested on NFSv4 share with ext4, xfs - FAIL

Tested NFSv4 share sync, async       - FAIL
Tested on NFSv3 share with xfs       - WORK
KVM with qemu rhel-6.5, 2.1.1        - SAME RESULTS
Quoted 7 lines

NOTE: it seems just the rhel5 guest triggers it easily..

HTH, Z.
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Don't use 3.16.x kernels with NFS yet. There are known bugs in the read/write code that need a stable backport of a set of fixes that are in 3.17-rc.

Signature · 5 lines
Trond Myklebust

Linux NFS client maintainer, PrimaryData

[email protected]
#2 lore raw
Quoted 70 lines · Trond Myklebust
Quoted 61 lines · Zdenek Kaspar

Hello everyone, got this specific problem with RHEL5 guest under KVM hypervisor.

client/server kernel version: 3.16.2

/etc/exports: rw,async,no_subtree_check,no_root_squash /proc/mounts: nfs4 rw,relatime,vers=4.0,rsize=524288,wsize=524288,

              namlen=255,hard,proto=tcp,port=0,
              timeo=600,retrans=2,sec=sys,clientaddr=xxx.xxx.xxx.xxx,
              local_lock=none,addr=xxx.xxx.xxx.yyy 0 0

CONFIG_NFS_FS=m
# CONFIG_NFS_V2 is not set
CONFIG_NFS_V3=m
# CONFIG_NFS_V3_ACL is not set
CONFIG_NFS_V4=m
# CONFIG_NFS_SWAP is not set
# CONFIG_NFS_V4_1 is not set
# CONFIG_NFS_USE_LEGACY_DNS is not set
CONFIG_NFS_USE_KERNEL_DNS=y
CONFIG_NFSD=m
CONFIG_NFSD_V3=y
# CONFIG_NFSD_V3_ACL is not set
CONFIG_NFSD_V4=y
# CONFIG_NFSD_FAULT_INJECTION is not set
CONFIG_NFS_COMMON=y

CentOS-5.10-i386-bin-1of8.iso -- did not test everything on centos, behaved same as RHEL5. So let's use it as VM reproducer:

dd if=/dev/zero of=kvm-test1.img bs=1M count=2048

start VM using:
file=kvm-test1.img,cache=writeback,aio=native,if=virtio,boot=on

create custom layout: /dev/vda1 / ext3 max size primary unselect Desktop-Gnome, select Customize now use only: Base System/Base and install it..

[root@test1 ~]# find / >/dev/null
everything works as expected with writeback

shutdown, backup kvm-test1.img (fs will explode) start VM using:
file=kvm-test1.img,cache=none,aio=native,if=virtio,boot=on

[root@test1 ~]# find / >/dev/null
and check dmesg, reboot etc, ie:
EXT3-fs error (device vda1): htree_dirblock_to_tree: bad entry in directory #460317: rec_len is smaller than minimal - offset=0, inode=0, rec_len=0, name_len=0
EXT3-fs error (device vda1): htree_dirblock_to_tree: bad entry in directory #98503: rec_len is smaller than minimal - offset=0, inode=0, rec_len=0, name_len=0

Tested locally on ext4, xfs - WORK
Tested on NFSv4 share with ext4, xfs - FAIL

Tested NFSv4 share sync, async       - FAIL
Tested on NFSv3 share with xfs       - WORK
KVM with qemu rhel-6.5, 2.1.1        - SAME RESULTS
Quoted 7 lines

NOTE: it seems just the rhel5 guest triggers it easily..

HTH, Z.
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 5 lines

Don't use 3.16.x kernels with NFS yet. There are known bugs in the read/write code that need a stable backport of a set of fixes that are in 3.17-rc.

Hi Trond, noticed big nfs update that got into -stable, running 3.16.4 on both sides and problem is fixed, thx!

Z.

#3 lore raw
Quoted 77 lines · Zdenek Kaspar
Quoted 70 lines · Trond Myklebust
Quoted 61 lines · Zdenek Kaspar

Hello everyone, got this specific problem with RHEL5 guest under KVM hypervisor.

client/server kernel version: 3.16.2

/etc/exports: rw,async,no_subtree_check,no_root_squash /proc/mounts: nfs4 rw,relatime,vers=4.0,rsize=524288,wsize=524288,

              namlen=255,hard,proto=tcp,port=0,
              timeo=600,retrans=2,sec=sys,clientaddr=xxx.xxx.xxx.xxx,
              local_lock=none,addr=xxx.xxx.xxx.yyy 0 0

CONFIG_NFS_FS=m
# CONFIG_NFS_V2 is not set
CONFIG_NFS_V3=m
# CONFIG_NFS_V3_ACL is not set
CONFIG_NFS_V4=m
# CONFIG_NFS_SWAP is not set
# CONFIG_NFS_V4_1 is not set
# CONFIG_NFS_USE_LEGACY_DNS is not set
CONFIG_NFS_USE_KERNEL_DNS=y
CONFIG_NFSD=m
CONFIG_NFSD_V3=y
# CONFIG_NFSD_V3_ACL is not set
CONFIG_NFSD_V4=y
# CONFIG_NFSD_FAULT_INJECTION is not set
CONFIG_NFS_COMMON=y

CentOS-5.10-i386-bin-1of8.iso -- did not test everything on centos, behaved same as RHEL5. So let's use it as VM reproducer:

dd if=/dev/zero of=kvm-test1.img bs=1M count=2048

start VM using:
file=kvm-test1.img,cache=writeback,aio=native,if=virtio,boot=on

create custom layout: /dev/vda1 / ext3 max size primary unselect Desktop-Gnome, select Customize now use only: Base System/Base and install it..

[root@test1 ~]# find / >/dev/null
everything works as expected with writeback

shutdown, backup kvm-test1.img (fs will explode) start VM using:
file=kvm-test1.img,cache=none,aio=native,if=virtio,boot=on

[root@test1 ~]# find / >/dev/null
and check dmesg, reboot etc, ie:
EXT3-fs error (device vda1): htree_dirblock_to_tree: bad entry in directory #460317: rec_len is smaller than minimal - offset=0, inode=0, rec_len=0, name_len=0
EXT3-fs error (device vda1): htree_dirblock_to_tree: bad entry in directory #98503: rec_len is smaller than minimal - offset=0, inode=0, rec_len=0, name_len=0

Tested locally on ext4, xfs - WORK
Tested on NFSv4 share with ext4, xfs - FAIL

Tested NFSv4 share sync, async       - FAIL
Tested on NFSv3 share with xfs       - WORK
KVM with qemu rhel-6.5, 2.1.1        - SAME RESULTS
Quoted 7 lines

NOTE: it seems just the rhel5 guest triggers it easily..

HTH, Z.
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 5 lines

Don't use 3.16.x kernels with NFS yet. There are known bugs in the read/write code that need a stable backport of a set of fixes that are in 3.17-rc.

Quoted 4 lines

Hi Trond, noticed big nfs update that got into -stable, running 3.16.4 on both sides and problem is fixed, thx!

Z.

Cool! Thanks for having tested and for confirming.

Cheers
Trond

Signature · 5 lines
Trond Myklebust

Linux NFS client maintainer, PrimaryData

[email protected]