Skip to contentSkip to conversation

nfs stale filehandle issues with 2.6.10-rc1 in-kernel server

DISCUSSION Brad Campbell 6 participants 25 messages Nov 2, 2004 – Nov 8, 2004 1 message not in the archive

lore
Share & tools
lore
#0 lore raw

G'day all,

While load testing for the page allocation failure errors, I managed to reasonably reliably reproduce an error that has been annoying me for a while with nfs.

I have 2 Machines. My server is running 2.6.10-rc1-BK as per a couple of days ago, but this has happened with 2.6.9-pre3 also.

Server :
Kernel - 2.6.10-rc1-BK
IP - 192.168.2.82
Interface - eth1 to Client. eth0 idle

Client :
Kernel - 2.4.26 (Highmem enabled)
IP - 192.168.2.81
Interface - eth0 to Server. eth1 to world

MTU == 9000
NFS mounted using tcp with w/rsize == 32768

The client boots of a usb keystick with loadlin and runs an NFS root from the server.

The issue is I randomly get files just return with "Stale NFS file handle"
merely doing a cd .. ; cd $OLDPWD will make them work again. Sometimes they come back by themselves.

I'm running Debian Sarge on both machines and using the kernel-nfs server compiled in rather than modular.

Using the same client, unmodified with a 2.6.5 kernel on the server, I never saw any of these issues in over three months of heavy use.

brad@tv:/raid2$ while true ; do dd if=/dev/zero of=test.1 bs=1M count=1000 ; done
dd: writing `test.1': Stale NFS file handle
dd: closing output file `test.1': Stale NFS file handle
dd: writing `test.1': Stale NFS file handle
dd: closing output file `test.1': Stale NFS file handle

Am I doing something stupid? Have I included all relevant information?

Regards,
Brad

--------------------Server Files-----------------------------

brad@srv:/raid/tobackup/dvd$ uname -a
Linux srv 2.6.10-rc1 #4 Tue Nov 2 14:55:49 GST 2004 i686 GNU/Linux
brad@srv:~$ cat /proc/cmdline
vga=1 console=ttyS0,38400 console=tty0 root=/dev/hda1
brad@srv:/raid/tobackup/dvd$ cat /etc/exports
# /etc/exports: the access control list for filesystems which may be exported
#               to NFS clients.  See exports(5).

/raid 192.168.2.81(rw,async,no_root_squash)
/raid 192.168.3.80(rw,async,no_root_squash)
/raid0 192.168.2.81(rw,async,no_root_squash)
/raid0/tmp 192.168.2.81(rw,async,no_root_squash)
/raid2 192.168.2.81(rw,async,no_root_squash)
/raid2 192.168.3.80(rw,async,no_root_squash)
/nfsroot 192.168.2.81(rw,async,no_root_squash)

brad@srv:~$ mount
/dev/hda1 on / type ext3 (rw,errors=remount-ro)
proc on /proc type proc (rw)
/sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
/dev/hda3 on /raid0 type ext3 (rw)
none on /proc/fs/nfsd type nfsd (rw)
none on /sysfs type sysfs (rw)
/dev/md0 on /raid type ext3 (rw)
/dev/md2 on /raid2 type ext3 (rw)
brad@srv:~$ cat /etc/hosts
127.0.0.1       srv     localhost
192.168.2.81    tv
192.168.3.80    bklaptop1
brad@srv:/raid/tobackup/dvd$ exportfs
/raid0/nfsroot  tv
/raid0/tmp      tv
/raid0          tv
/raid2          tv
/raid2          bklaptop1
/raid           tv
/raid           bklaptop1
brad@srv:/raid/tobackup/dvd$ ifconfig
eth0      Link encap:Ethernet  HWaddr 00:0E:A6:41:45:94
          inet addr:192.168.3.82  Bcast:192.168.3.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)
          Interrupt:18 Memory:de800000-0

eth1 Link encap:Ethernet HWaddr 00:04:E2:8E:1E:AD

          inet addr:192.168.2.82  Bcast:192.168.2.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:9000  Metric:1
          RX packets:15403785 errors:0 dropped:0 overruns:0 frame:0
          TX packets:14267922 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1831885829 (1.7 GiB)  TX bytes:3547388866 (3.3 GiB)
          Interrupt:16 Memory:dc800000-0

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:66 errors:0 dropped:0 overruns:0 frame:0 TX packets:66 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0
RX bytes:4600 (4.4 KiB) TX bytes:4600 (4.4 KiB)

brad@srv:~$ lspci
0000:00:00.0 Host bridge: VIA Technologies, Inc. VT8377 [KT400/KT600 AGP] Host Bridge (rev 80)
0000:00:01.0 PCI bridge: VIA Technologies, Inc. VT8237 PCI Bridge
0000:00:09.0 Ethernet controller: 3Com Corporation 3c940 10/100/1000Base-T [Marvell] (rev 12)
0000:00:0b.0 Unknown mass storage controller: Promise Technology, Inc. PDC20318 (SATA150 TX4) (rev 02)
0000:00:0c.0 FireWire (IEEE 1394): VIA Technologies, Inc. IEEE 1394 Host Controller (rev 46)
0000:00:0d.0 Ethernet controller: Marvell Technology Group Ltd. Yukon Gigabit Ethernet 10/100/1000Base-T Adapter (rev 12)
0000:00:0e.0 Unknown mass storage controller: Promise Technology, Inc. PDC20318 (SATA150 TX4) (rev 02)
0000:00:0f.0 RAID bus controller: VIA Technologies, Inc. VIA VT6420 SATA RAID Controller (rev 80)
0000:00:0f.1 IDE interface: VIA Technologies, Inc. VT82C586A/B/VT82C686/A/B/VT823x/A/C PIPC Bus Master IDE (rev 06)
0000:00:10.0 USB Controller: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller (rev 81)
0000:00:10.1 USB Controller: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller (rev 81)
0000:00:10.2 USB Controller: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller (rev 81)
0000:00:10.3 USB Controller: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller (rev 81)
0000:00:10.4 USB Controller: VIA Technologies, Inc. USB 2.0 (rev 86)
0000:00:11.0 ISA bridge: VIA Technologies, Inc. VT8237 ISA bridge [K8T800 South]
0000:00:13.0 Unknown mass storage controller: Promise Technology, Inc. PDC20318 (SATA150 TX4) (rev 02)
0000:01:00.0 VGA compatible controller: Silicon Integrated Systems [SiS] 315PRO PCI/AGP VGA Display Adapter
brad@srv:~$ lsmod
Module                  Size  Used by
ehci_hcd               39492  0
uhci_hcd               30988  0
usbcore               118072  3 ehci_hcd,uhci_hcd
it87                   22124  0
i2c_sensor              3072  1 it87
i2c_isa                 1728  0
ohci1394               31556  0
ieee1394              307764  1 ohci1394
sk98lin               167016  2

CONFIG_X86=y
CONFIG_MMU=y
CONFIG_UID16=y
CONFIG_GENERIC_ISA_DMA=y
CONFIG_GENERIC_IOMAP=y
CONFIG_EXPERIMENTAL=y
CONFIG_CLEAN_COMPILE=y
CONFIG_BROKEN_ON_SMP=y
CONFIG_LOCK_KERNEL=y
CONFIG_LOCALVERSION=""
CONFIG_SWAP=y
CONFIG_SYSVIPC=y
CONFIG_POSIX_MQUEUE=y
CONFIG_SYSCTL=y
CONFIG_LOG_BUF_SHIFT=20
CONFIG_HOTPLUG=y
CONFIG_KOBJECT_UEVENT=y
CONFIG_IKCONFIG=y
CONFIG_IKCONFIG_PROC=y
CONFIG_KALLSYMS=y
CONFIG_KALLSYMS_ALL=y
CONFIG_KALLSYMS_EXTRA_PASS=y
CONFIG_FUTEX=y
CONFIG_EPOLL=y
CONFIG_SHMEM=y
CONFIG_CC_ALIGN_FUNCTIONS=0
CONFIG_CC_ALIGN_LABELS=0
CONFIG_CC_ALIGN_LOOPS=0
CONFIG_CC_ALIGN_JUMPS=0
CONFIG_MODULES=y
CONFIG_MODULE_UNLOAD=y
CONFIG_OBSOLETE_MODPARM=y
CONFIG_KMOD=y
CONFIG_X86_PC=y
CONFIG_MK7=y
CONFIG_X86_CMPXCHG=y
CONFIG_X86_XADD=y
CONFIG_X86_L1_CACHE_SHIFT=6
CONFIG_RWSEM_XCHGADD_ALGORITHM=y
CONFIG_X86_WP_WORKS_OK=y
CONFIG_X86_INVLPG=y
CONFIG_X86_BSWAP=y
CONFIG_X86_POPAD_OK=y
CONFIG_X86_GOOD_APIC=y
CONFIG_X86_INTEL_USERCOPY=y
CONFIG_X86_USE_PPRO_CHECKSUM=y
CONFIG_X86_USE_3DNOW=y
CONFIG_HPET_TIMER=y
CONFIG_HPET_EMULATE_RTC=y
CONFIG_PREEMPT=y
CONFIG_X86_UP_APIC=y
CONFIG_X86_UP_IOAPIC=y
CONFIG_X86_LOCAL_APIC=y
CONFIG_X86_IO_APIC=y
CONFIG_X86_TSC=y
CONFIG_X86_MCE=y
CONFIG_X86_MCE_NONFATAL=y
CONFIG_NOHIGHMEM=y
CONFIG_MTRR=y
CONFIG_HAVE_DEC_LOCK=y
CONFIG_PM=y
CONFIG_ACPI=y
CONFIG_ACPI_BOOT=y
CONFIG_ACPI_INTERPRETER=y
CONFIG_ACPI_BUTTON=y
CONFIG_ACPI_BLACKLIST_YEAR=0
CONFIG_ACPI_BUS=y
CONFIG_ACPI_EC=y
CONFIG_ACPI_POWER=y
CONFIG_ACPI_PCI=y
CONFIG_ACPI_SYSTEM=y
CONFIG_PCI=y
CONFIG_PCI_GOBIOS=y
CONFIG_PCI_BIOS=y
CONFIG_PCI_LEGACY_PROC=y
CONFIG_PCI_NAMES=y
CONFIG_BINFMT_ELF=y
CONFIG_BINFMT_AOUT=m
CONFIG_BINFMT_MISC=m
CONFIG_STANDALONE=y
CONFIG_PREVENT_FIRMWARE_BUILD=y
CONFIG_INITRAMFS_SOURCE=""
CONFIG_LBD=y
CONFIG_IOSCHED_NOOP=y
CONFIG_IOSCHED_AS=y
CONFIG_IOSCHED_DEADLINE=y
CONFIG_IOSCHED_CFQ=y
CONFIG_IDE=y
CONFIG_BLK_DEV_IDE=y
CONFIG_BLK_DEV_IDEDISK=y
CONFIG_IDEDISK_MULTI_MODE=y
CONFIG_BLK_DEV_IDEPCI=y
CONFIG_IDEPCI_SHARE_IRQ=y
CONFIG_BLK_DEV_IDEDMA_PCI=y
CONFIG_IDEDMA_PCI_AUTO=y
CONFIG_BLK_DEV_VIA82CXXX=y
CONFIG_BLK_DEV_IDEDMA=y
CONFIG_IDEDMA_AUTO=y
CONFIG_SCSI=y
CONFIG_SCSI_PROC_FS=y
CONFIG_BLK_DEV_SD=y
CONFIG_BLK_DEV_SR=y
CONFIG_CHR_DEV_SG=y
CONFIG_SCSI_CONSTANTS=y
CONFIG_SCSI_LOGGING=y
CONFIG_SCSI_SATA=y
CONFIG_SCSI_SATA_PROMISE=y
CONFIG_SCSI_SATA_VIA=y
CONFIG_SCSI_QLA2XXX=y
CONFIG_MD=y
CONFIG_BLK_DEV_MD=y
CONFIG_MD_RAID0=y
CONFIG_MD_RAID5=y
CONFIG_MD_RAID6=y
CONFIG_IEEE1394=m
CONFIG_IEEE1394_OUI_DB=y
CONFIG_IEEE1394_OHCI1394=m
CONFIG_IEEE1394_SBP2=m
CONFIG_IEEE1394_RAWIO=m
CONFIG_NET=y
CONFIG_PACKET=y
CONFIG_UNIX=y
CONFIG_INET=y
CONFIG_IP_MULTICAST=y
CONFIG_BRIDGE=y
CONFIG_NETDEVICES=y
CONFIG_DUMMY=m
CONFIG_SK98LIN=m
CONFIG_TIGON3=m
CONFIG_INPUT=y
CONFIG_INPUT_MOUSEDEV=y
CONFIG_INPUT_MOUSEDEV_PSAUX=y
CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024
CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768
CONFIG_SOUND_GAMEPORT=y
CONFIG_SERIO=y
CONFIG_SERIO_I8042=y
CONFIG_INPUT_KEYBOARD=y
CONFIG_KEYBOARD_ATKBD=y
CONFIG_INPUT_MOUSE=y
CONFIG_MOUSE_PS2=y
CONFIG_VT=y
CONFIG_VT_CONSOLE=y
CONFIG_HW_CONSOLE=y
CONFIG_SERIAL_8250=y
CONFIG_SERIAL_8250_CONSOLE=y
CONFIG_SERIAL_8250_NR_UARTS=4
CONFIG_SERIAL_CORE=y
CONFIG_SERIAL_CORE_CONSOLE=y
CONFIG_UNIX98_PTYS=y
CONFIG_LEGACY_PTYS=y
CONFIG_LEGACY_PTY_COUNT=256
CONFIG_RTC=y
CONFIG_HPET=y
CONFIG_HPET_MMAP=y
CONFIG_I2C=y
CONFIG_I2C_CHARDEV=y
CONFIG_I2C_ALGOBIT=y
CONFIG_I2C_ISA=m
CONFIG_I2C_SENSOR=m
CONFIG_SENSORS_IT87=m
CONFIG_VIDEO_SELECT=y
CONFIG_VGA_CONSOLE=y
CONFIG_DUMMY_CONSOLE=y
CONFIG_USB=m
CONFIG_USB_DEBUG=y
CONFIG_USB_DEVICEFS=y
CONFIG_USB_ARCH_HAS_HCD=y
CONFIG_USB_ARCH_HAS_OHCI=y
CONFIG_USB_EHCI_HCD=m
CONFIG_USB_UHCI_HCD=m
CONFIG_USB_STORAGE=m
CONFIG_USB_HID=m
CONFIG_USB_HIDINPUT=y
CONFIG_EXT2_FS=y
CONFIG_EXT3_FS=y
CONFIG_EXT3_FS_XATTR=y
CONFIG_JBD=y
CONFIG_FS_MBCACHE=y
CONFIG_FS_POSIX_ACL=y
CONFIG_DNOTIFY=y
CONFIG_ISO9660_FS=y
CONFIG_JOLIET=y
CONFIG_UDF_FS=y
CONFIG_UDF_NLS=y
CONFIG_FAT_FS=y
CONFIG_MSDOS_FS=y
CONFIG_VFAT_FS=y
CONFIG_FAT_DEFAULT_CODEPAGE=437
CONFIG_FAT_DEFAULT_IOCHARSET="iso8859-1"
CONFIG_PROC_FS=y
CONFIG_PROC_KCORE=y
CONFIG_SYSFS=y
CONFIG_DEVFS_FS=y
CONFIG_TMPFS=y
CONFIG_RAMFS=y
CONFIG_NFS_FS=y
CONFIG_NFS_V3=y
CONFIG_NFS_V4=y
CONFIG_NFS_DIRECTIO=y
CONFIG_NFSD=y
CONFIG_NFSD_V3=y
CONFIG_NFSD_V4=y
CONFIG_NFSD_TCP=y
CONFIG_LOCKD=y
CONFIG_LOCKD_V4=y
CONFIG_EXPORTFS=y
CONFIG_SUNRPC=y
CONFIG_SUNRPC_GSS=y
CONFIG_RPCSEC_GSS_KRB5=y
CONFIG_MSDOS_PARTITION=y
CONFIG_NLS=y
CONFIG_NLS_DEFAULT="iso8859-1"
CONFIG_NLS_CODEPAGE_437=y
CONFIG_NLS_ISO8859_1=y
CONFIG_DEBUG_KERNEL=y
CONFIG_MAGIC_SYSRQ=y
CONFIG_DEBUG_SPINLOCK_SLEEP=y
CONFIG_EARLY_PRINTK=y
CONFIG_X86_FIND_SMP_CONFIG=y
CONFIG_X86_MPPARSE=y
CONFIG_CRYPTO=y
CONFIG_CRYPTO_MD5=y
CONFIG_CRYPTO_DES=y
CONFIG_CRC32=y
CONFIG_GENERIC_HARDIRQS=y
CONFIG_GENERIC_IRQ_PROBE=y
CONFIG_X86_BIOS_REBOOT=y
CONFIG_PC=y

--------------------Client Files------------------------------

brad@tv:/raid2$ uname -a
Linux tv 2.4.26-hm #2 Mon Aug 23 18:13:02 GST 2004 i686 GNU/Linux
brad@tv:~$ cat /proc/cmdline
root=/dev/nfs video=matrox:xres:1024,yres:768,pixclock:15386,left:160,right:24,upper:29,lower:3,hslen:132,vslen:6,depth:8 nfsroot=192.168.2.82:/nfsroot,rsize=32768,wsize=32768,tcp ip=192.168.2.81:192.168.2.82::255.255.255.0:tv clock=pit panic=15 BOOT_IMAG
brad@tv:/raid2$ cat /etc/fstab
# /etc/fstab: static file system information.
#
# <file system> <mount point>   <type>  <options>               <dump>  <pass>
192.168.2.82:/nfsroot   / nfs   rw                              1       1
proc            /proc           proc    defaults                0       0
none            /dev/devfs      devfs   defaults
192.168.2.82:/raid /raid        nfs     rsize=32768,wsize=32768,tcp     0      0
192.168.2.82:/raid0/tmp /raid0  nfs     rsize=32768,wsize=32768,tcp     0      0
none  /proc/bus/usb  usbdevfs  defaults                           0      0
192.168.2.82:/raid2 /raid2      nfs     rsize=32768,wsize=32768,tcp     0      0
none    /tmp    tmpfs   defaults,size=512M                        0      0
brad@tv:/raid2$ mount
192.168.2.82:/nfsroot on / type nfs (rw)
proc on /proc type proc (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
none on /dev/devfs type devfs (rw)
none on /proc/bus/usb type usbdevfs (rw)
none on /tmp type tmpfs (rw,size=512M)
192.168.2.82:/raid on /raid type nfs (rw,rsize=32768,wsize=32768,tcp,addr=192.168.2.82)
192.168.2.82:/raid0/tmp on /raid0 type nfs (rw,rsize=32768,wsize=32768,tcp,addr=192.168.2.82)
192.168.2.82:/raid2 on /raid2 type nfs (rw,rsize=32768,wsize=32768,tcp,addr=192.168.2.82)
brad@tv:/raid2$ ifconfig
eth0      Link encap:Ethernet  HWaddr 00:04:E2:8E:1E:65
          inet addr:192.168.2.81  Bcast:192.168.2.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:55678032 errors:0 dropped:0 overruns:0 frame:0
          TX packets:68316272 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:4155793385 (3.8 GiB)  TX bytes:3579805141 (3.3 GiB)
          Interrupt:17 Memory:dd000000-0

eth1 Link encap:Ethernet HWaddr 00:05:5D:5B:C5:4E

          inet addr:192.168.2.81  Bcast:192.168.2.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:2182660 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2465204 errors:17 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:995104632 (949.0 MiB)  TX bytes:1197365443 (1.1 GiB)
          Interrupt:17 Memory:dd008000-dd008fff

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:27 errors:0 dropped:0 overruns:0 frame:0 TX packets:27 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0
RX bytes:1908 (1.8 KiB) TX bytes:1908 (1.8 KiB)

brad@tv:~$ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
192.168.2.82    0.0.0.0         255.255.255.255 UH    0      0        0 eth0
192.168.2.0     0.0.0.0         255.255.255.0   U     0      0        0 eth1
0.0.0.0         192.168.2.3     0.0.0.0         UG    0      0        0 eth1
brad@tv:/usr/src/linux$ lspci
0000:00:00.0 Host bridge: VIA Technologies, Inc. VT8377 [KT400/KT600 AGP] Host Bridge (rev 80)
0000:00:01.0 PCI bridge: VIA Technologies, Inc. VT8237 PCI Bridge
0000:00:09.0 Network controller: Intersil Corporation Prism 2.5 Wavelan chipset (rev 01)
0000:00:0a.0 Multimedia video controller: Brooktree Corporation Bt878 Video Capture (rev 11)
0000:00:0a.1 Multimedia controller: Brooktree Corporation Bt878 Audio Capture (rev 11)
0000:00:0b.0 Multimedia audio controller: Creative Labs SB Live! EMU10k1 (rev 0a)
0000:00:0b.1 Input device controller: Creative Labs SB Live! MIDI/Game Port (rev 0a)
0000:00:0c.0 Multimedia video controller: Brooktree Corporation Bt878 Video Capture (rev 11)
0000:00:0c.1 Multimedia controller: Brooktree Corporation Bt878 Audio Capture (rev 11)
0000:00:0d.0 Ethernet controller: Marvell Technology Group Ltd. Yukon Gigabit Ethernet 10/100/1000Base-T Adapter (rev 12)
0000:00:0f.0 IDE interface: VIA Technologies, Inc. VT82C586A/B/VT82C686/A/B/VT823x/A/C PIPC Bus Master IDE (rev 06)
0000:00:10.0 USB Controller: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller (rev 81)
0000:00:10.1 USB Controller: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller (rev 81)
0000:00:10.2 USB Controller: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller (rev 81)
0000:00:10.4 USB Controller: VIA Technologies, Inc. USB 2.0 (rev 86)
0000:00:11.0 ISA bridge: VIA Technologies, Inc. VT8237 ISA bridge [K8T800 South]
0000:00:11.5 Multimedia audio controller: VIA Technologies, Inc. VT8233/A/8235/8237 AC97 Audio Controller (rev 60)
0000:01:00.0 VGA compatible controller: Matrox Graphics, Inc. MGA G400 AGP (rev 82)
brad@tv:~$ lsmod
Module                  Size  Used by    Not tainted
iptable_filter          1772   0 (autoclean) (unused)
ide-cd                 32096   0 (autoclean)
cdrom                  28352   0 (autoclean) [ide-cd]
isofs                  27124   0 (autoclean)
zlib_inflate           18308   0 (autoclean) [isofs]
lirc_serial             7296   0 (unused)
ipt_MASQUERADE          1400   0 (autoclean)
iptable_nat            16814   0 (autoclean) [ipt_MASQUERADE]
ip_conntrack           20004   0 (autoclean) [ipt_MASQUERADE iptable_nat]
ip_tables              12416   5 [iptable_filter ipt_MASQUERADE iptable_nat]
nbd                    15300   0 (autoclean) (unused)
usb-storage            26512   0 (unused)
scsi_mod               86692   1 [usb-storage]
mousedev                4148   1
uhci                   25372   0 (unused)
ehci-hcd               18156   0 (unused)
snd-via82xx            14180   0
snd-mpu401-uart         3488   0 [snd-via82xx]
tun                     4256   0 (unused)
ide-detect               288   0 (unused)
via82cxxx              10696   1
ide-core              106556   0 [ide-cd ide-detect via82cxxx]
dvb-bt8xx               4220   0 (unused)
dst                    10612   1
bt878                   5424   0 [dvb-bt8xx dst]
dvb-core               41760   0 [dvb-bt8xx dst]
tuner                  14252   1 (autoclean)
bttv                  115372   0 [dvb-bt8xx bt878]
video-buf              11888   0 [bttv]
btcx-risc               2328   0 [bttv]
i2c-algo-bit            7464   0 [bttv]
i2c-core               14820   0 [dvb-bt8xx tuner bttv i2c-algo-bit]
videodev                4416   3 [bttv]
v4l2-common             3168   0 [bttv]
lirc_streamzap          7036   0
lirc_dev                8756   2 [lirc_serial lirc_streamzap]
uinput                  4000   0
joydev                  5952   0 (unused)
keybdev                 2212   0 (unused)
hid                    21764   0 (unused)
input                   3328   0 [mousedev uinput joydev keybdev hid]
snd-usb-audio          43136   0
snd-usb-lib             8612   0 [snd-usb-audio]
usbcore                62380   1 [usb-storage uhci ehci-hcd lirc_streamzap hid snd-usb-audio snd-usb-lib]
snd-emu10k1            76904   0
snd-pcm-oss            37928   0
snd-mixer-oss          13688   0 [snd-pcm-oss]
snd-hwdep               5252   0 [snd-emu10k1]
snd-util-mem            1296   0 [snd-emu10k1]
snd-ac97-codec         57728   0 [snd-via82xx snd-emu10k1]
snd-pcm                59624   0 [snd-via82xx snd-usb-audio snd-emu10k1 snd-pcm-oss snd-ac97-codec]
snd-timer              14596   0 [snd-pcm]
snd-rawmidi            13572   0 [snd-mpu401-uart snd-usb-lib snd-emu10k1]
snd-seq-device          4436   0 [snd-emu10k1 snd-rawmidi]
snd                    34820   0 [snd-via82xx snd-mpu401-uart snd-usb-audio snd-usb-lib snd-emu10k1 snd-pcm-oss snd-mixer-oss snd-hwdep snd-util-mem snd-ac97-codec snd-pcm snd-timer snd-rawmidi snd-seq-device]
soundcore               3780  10 [snd]
snd-page-alloc          5260   0 [snd-via82xx snd-usb-audio snd-emu10k1 snd-mixer-oss snd-hwdep snd-pcm snd-timer snd-rawmidi snd-seq-device snd]
orinoco_pci             3268   1
orinoco                34836   0 [orinoco_pci]
hermes                  5572   0 [orinoco_pci orinoco]
mga_vid                 9952   0 (unused)
rtc                     7048   0 (autoclean)

CONFIG_X86=y
CONFIG_UID16=y
CONFIG_EXPERIMENTAL=y
CONFIG_MODULES=y
CONFIG_KMOD=y
CONFIG_MK7=y
CONFIG_X86_WP_WORKS_OK=y
CONFIG_X86_INVLPG=y
CONFIG_X86_CMPXCHG=y
CONFIG_X86_XADD=y
CONFIG_X86_BSWAP=y
CONFIG_X86_POPAD_OK=y
CONFIG_RWSEM_XCHGADD_ALGORITHM=y
CONFIG_X86_L1_CACHE_SHIFT=6
CONFIG_X86_HAS_TSC=y
CONFIG_X86_GOOD_APIC=y
CONFIG_X86_USE_3DNOW=y
CONFIG_X86_PGE=y
CONFIG_X86_USE_PPRO_CHECKSUM=y
CONFIG_X86_F00F_WORKS_OK=y
CONFIG_X86_MCE=y
CONFIG_HIGHMEM4G=y
CONFIG_HIGHMEM=y
CONFIG_MTRR=y
CONFIG_X86_UP_APIC=y
CONFIG_X86_UP_IOAPIC=y
CONFIG_X86_LOCAL_APIC=y
CONFIG_X86_IO_APIC=y
CONFIG_X86_TSC=y
CONFIG_NET=y
CONFIG_PCI=y
CONFIG_PCI_GOANY=y
CONFIG_PCI_BIOS=y
CONFIG_PCI_DIRECT=y
CONFIG_ISA=y
CONFIG_PCI_NAMES=y
CONFIG_HOTPLUG=y
CONFIG_SYSVIPC=y
CONFIG_SYSCTL=y
CONFIG_KCORE_ELF=y
CONFIG_BINFMT_AOUT=y
CONFIG_BINFMT_ELF=y
CONFIG_BINFMT_MISC=y
CONFIG_PM=y
CONFIG_ACPI=y
CONFIG_ACPI_BOOT=y
CONFIG_ACPI_BUS=y
CONFIG_ACPI_INTERPRETER=y
CONFIG_ACPI_EC=y
CONFIG_ACPI_POWER=y
CONFIG_ACPI_PCI=y
CONFIG_ACPI_SLEEP=y
CONFIG_ACPI_SYSTEM=y
CONFIG_ACPI_BUTTON=y
CONFIG_ACPI_PROCESSOR=y
CONFIG_ACPI_THERMAL=y
CONFIG_BLK_DEV_LOOP=m
CONFIG_BLK_DEV_NBD=m
CONFIG_PACKET=y
CONFIG_NETFILTER=y
CONFIG_FILTER=y
CONFIG_UNIX=y
CONFIG_INET=y
CONFIG_IP_MULTICAST=y
CONFIG_IP_PNP=y
CONFIG_IP_NF_CONNTRACK=m
CONFIG_IP_NF_FTP=m
CONFIG_IP_NF_AMANDA=m
CONFIG_IP_NF_TFTP=m
CONFIG_IP_NF_IRC=m
CONFIG_IP_NF_IPTABLES=m
CONFIG_IP_NF_MATCH_LIMIT=m
CONFIG_IP_NF_MATCH_MAC=m
CONFIG_IP_NF_MATCH_PKTTYPE=m
CONFIG_IP_NF_MATCH_MARK=m
CONFIG_IP_NF_MATCH_MULTIPORT=m
CONFIG_IP_NF_MATCH_TOS=m
CONFIG_IP_NF_MATCH_RECENT=m
CONFIG_IP_NF_MATCH_ECN=m
CONFIG_IP_NF_MATCH_DSCP=m
CONFIG_IP_NF_MATCH_AH_ESP=m
CONFIG_IP_NF_MATCH_LENGTH=m
CONFIG_IP_NF_MATCH_TTL=m
CONFIG_IP_NF_MATCH_TCPMSS=m
CONFIG_IP_NF_MATCH_HELPER=m
CONFIG_IP_NF_MATCH_STATE=m
CONFIG_IP_NF_MATCH_CONNTRACK=m
CONFIG_IP_NF_MATCH_UNCLEAN=m
CONFIG_IP_NF_MATCH_OWNER=m
CONFIG_IP_NF_FILTER=m
CONFIG_IP_NF_TARGET_REJECT=m
CONFIG_IP_NF_TARGET_MIRROR=m
CONFIG_IP_NF_NAT=m
CONFIG_IP_NF_NAT_NEEDED=y
CONFIG_IP_NF_TARGET_MASQUERADE=m
CONFIG_IP_NF_TARGET_REDIRECT=m
CONFIG_IP_NF_NAT_AMANDA=m
CONFIG_IP_NF_NAT_LOCAL=y
CONFIG_IP_NF_NAT_SNMP_BASIC=m
CONFIG_IP_NF_NAT_IRC=m
CONFIG_IP_NF_NAT_FTP=m
CONFIG_IP_NF_NAT_TFTP=m
CONFIG_IP_NF_MANGLE=m
CONFIG_IP_NF_TARGET_TOS=m
CONFIG_IP_NF_TARGET_ECN=m
CONFIG_IP_NF_TARGET_DSCP=m
CONFIG_IP_NF_TARGET_MARK=m
CONFIG_IP_NF_TARGET_LOG=m
CONFIG_IP_NF_TARGET_ULOG=m
CONFIG_IP_NF_TARGET_TCPMSS=m
CONFIG_IP_NF_ARPTABLES=m
CONFIG_IP_NF_ARPFILTER=m
CONFIG_IP_NF_ARP_MANGLE=m
CONFIG_IDE=m
CONFIG_BLK_DEV_IDE=m
CONFIG_BLK_DEV_IDEDISK=m
CONFIG_BLK_DEV_IDECD=m
CONFIG_BLK_DEV_IDEPCI=y
CONFIG_IDEPCI_SHARE_IRQ=y
CONFIG_BLK_DEV_IDEDMA_PCI=y
CONFIG_IDEDMA_PCI_AUTO=y
CONFIG_BLK_DEV_IDEDMA=y
CONFIG_BLK_DEV_VIA82CXXX=m
CONFIG_IDEDMA_AUTO=y
CONFIG_SCSI=m
CONFIG_BLK_DEV_SD=m
CONFIG_SD_EXTRA_DEVS=40
CONFIG_BLK_DEV_SR=m
CONFIG_SR_EXTRA_DEVS=2
CONFIG_CHR_DEV_SG=m
CONFIG_SCSI_DEBUG_QUEUES=y
CONFIG_SCSI_MULTI_LUN=y
CONFIG_SCSI_CONSTANTS=y
CONFIG_IEEE1394=m
CONFIG_IEEE1394_OHCI1394=m
CONFIG_IEEE1394_SBP2=m
CONFIG_IEEE1394_OUI_DB=y
CONFIG_NETDEVICES=y
CONFIG_DUMMY=m
CONFIG_TUN=m
CONFIG_NET_ETHERNET=y
CONFIG_NET_PCI=y
CONFIG_EEPRO100=y
CONFIG_SK98LIN=y
CONFIG_NET_RADIO=y
CONFIG_HERMES=m
CONFIG_PCI_HERMES=m
CONFIG_NET_WIRELESS=y
CONFIG_INPUT=m
CONFIG_INPUT_KEYBDEV=m
CONFIG_INPUT_MOUSEDEV=m
CONFIG_INPUT_MOUSEDEV_SCREEN_X=1024
CONFIG_INPUT_MOUSEDEV_SCREEN_Y=768
CONFIG_INPUT_JOYDEV=m
CONFIG_INPUT_EVDEV=m
CONFIG_INPUT_UINPUT=m
CONFIG_VT=y
CONFIG_VT_CONSOLE=y
CONFIG_SERIAL=y
CONFIG_UNIX98_PTYS=y
CONFIG_UNIX98_PTY_COUNT=256
CONFIG_I2C=m
CONFIG_I2C_ALGOBIT=m
CONFIG_I2C_PROC=m
CONFIG_MOUSE=y
CONFIG_PSMOUSE=y
CONFIG_HW_RANDOM=y
CONFIG_RTC=m
CONFIG_AGP=y
CONFIG_AGP_VIA=y
CONFIG_DRM=y
CONFIG_DRM_NEW=y
CONFIG_DRM_MGA=m
CONFIG_VIDEO_DEV=m
CONFIG_EXT3_FS=m
CONFIG_JBD=m
CONFIG_FAT_FS=m
CONFIG_VFAT_FS=m
CONFIG_TMPFS=y
CONFIG_RAMFS=y
CONFIG_ISO9660_FS=m
CONFIG_JOLIET=y
CONFIG_ZISOFS=y
CONFIG_NTFS_FS=m
CONFIG_PROC_FS=y
CONFIG_DEVFS_FS=y
CONFIG_DEVPTS_FS=y
CONFIG_EXT2_FS=m
CONFIG_NFS_FS=y
CONFIG_NFS_V3=y
CONFIG_NFS_DIRECTIO=y
CONFIG_ROOT_NFS=y
CONFIG_NFSD=m
CONFIG_NFSD_V3=y
CONFIG_NFSD_TCP=y
CONFIG_SUNRPC=y
CONFIG_LOCKD=y
CONFIG_LOCKD_V4=y
CONFIG_SMB_FS=m
CONFIG_SMB_NLS_DEFAULT=y
CONFIG_SMB_NLS_REMOTE="cp437"
CONFIG_ZISOFS_FS=m
CONFIG_MSDOS_PARTITION=y
CONFIG_SMB_NLS=y
CONFIG_NLS=y
CONFIG_NLS_DEFAULT="iso8859-1"
CONFIG_NLS_CODEPAGE_437=m
CONFIG_NLS_ISO8859_1=m
CONFIG_VGA_CONSOLE=y
CONFIG_VIDEO_SELECT=y
CONFIG_FB=y
CONFIG_DUMMY_CONSOLE=y
CONFIG_VIDEO_SELECT=y
CONFIG_FB_MATROX=y
CONFIG_FB_MATROX_G450=y
CONFIG_FB_MATROX_G100=y
CONFIG_FB_MATROX_PROC=y
CONFIG_FBCON_CFB8=y
CONFIG_FBCON_CFB16=y
CONFIG_FBCON_CFB24=y
CONFIG_FBCON_CFB32=y
CONFIG_FBCON_FONTS=y
CONFIG_FONT_8x8=y
CONFIG_FONT_8x16=y
CONFIG_SOUND=m
CONFIG_SOUND_VIA82CXXX=m
CONFIG_USB=m
CONFIG_USB_DEVICEFS=y
CONFIG_USB_EHCI_HCD=m
CONFIG_USB_UHCI_ALT=m
CONFIG_USB_STORAGE=m
CONFIG_USB_HID=m
CONFIG_USB_HIDINPUT=y
CONFIG_USB_HIDDEV=y
CONFIG_USB_USBNET=m
CONFIG_DEBUG_KERNEL=y
CONFIG_MAGIC_SYSRQ=y
CONFIG_LOG_BUF_SHIFT=0
CONFIG_CRC32=y
CONFIG_ZLIB_INFLATE=m
CONFIG_ZLIB_DEFLATE=y
CONFIG_FW_LOADER=y

#1 lore raw

ty den 02.11.2004 Klokka 16:02 (+0400) skreiv Brad Campbell:

Quoted 7 lines

/raid 192.168.2.81(rw,async,no_root_squash)
/raid 192.168.3.80(rw,async,no_root_squash)
/raid0 192.168.2.81(rw,async,no_root_squash)
/raid0/tmp 192.168.2.81(rw,async,no_root_squash)
/raid2 192.168.2.81(rw,async,no_root_squash)
/raid2 192.168.3.80(rw,async,no_root_squash)
/nfsroot 192.168.2.81(rw,async,no_root_squash)

You should only have 1 line per directory.

You are not allowed to export a directory and its child (unless the child is on a different partition - which does not appear to be the case here).

http://nfs.sourceforge.net/nfs-howto/server.html#CONFIG

Cheers,
Trond

Signature · 1 line
Trond Myklebust <[email protected]>
#2 lore raw
Quoted 12 lines · Trond Myklebust

ty den 02.11.2004 Klokka 16:02 (+0400) skreiv Brad Campbell:

Quoted 7 lines

/raid 192.168.2.81(rw,async,no_root_squash)
/raid 192.168.3.80(rw,async,no_root_squash)
/raid0 192.168.2.81(rw,async,no_root_squash)
/raid0/tmp 192.168.2.81(rw,async,no_root_squash)
/raid2 192.168.2.81(rw,async,no_root_squash)
/raid2 192.168.3.80(rw,async,no_root_squash)
/nfsroot 192.168.2.81(rw,async,no_root_squash)

Quoted 7 lines

You should only have 1 line per directory.

You are not allowed to export a directory and its child (unless the child is on a different partition - which does not appear to be the case here).

http://nfs.sourceforge.net/nfs-howto/server.html#CONFIG

I'm also seeing stale filehandle problems here in recent kernels.

Setup: x86 or x86-64, TCP, NFSv4 compiled in to both server and client, but not specified in mount options.

This is readily reproducible with rsync -- I just boot to an earlier version of the kernel on the NFS client, and the stale filehandle problems go away.

JEff

#3 lore raw
Quoted 1 line · Jeff Garzik

Trond Myklebust wrote:

...

Quoted 11 lines
Quoted 1 line

http://nfs.sourceforge.net/nfs-howto/server.html#CONFIG

I'm also seeing stale filehandle problems here in recent kernels.

Setup: x86 or x86-64, TCP, NFSv4 compiled in to both server and client, but not specified in mount options.

This is readily reproducible with rsync -- I just boot to an earlier version of the kernel on the NFS client, and the stale filehandle problems go away.

Hi Jeff,

Does running an 'ls' on the server in the exported directory that is stale on the client resolve the problem (temporarily)?

(working with really weird stale handle theories here I admit...)

Signature · 1 line
 / jakob
#4 lore raw
Quoted 13 lines · Jakob Oestergaard
Quoted 1 line · Jeff Garzik

Trond Myklebust wrote:

...

Quoted 7 lines
Quoted 1 line

http://nfs.sourceforge.net/nfs-howto/server.html#CONFIG

I'm also seeing stale filehandle problems here in recent kernels.

Setup: x86 or x86-64, TCP, NFSv4 compiled in to both server and client, but not specified in mount options.

Quoted 7 lines
Quoted 1 line

This is readily reproducible with rsync -- I just boot to an earlier version of the kernel on the NFS client, and the stale filehandle problems go away.

Hi Jeff,

Does running an 'ls' on the server in the exported directory that is stale on the client resolve the problem (temporarily)?

Yes.

Jeff

#5 lore raw

to den 04.11.2004 Klokka 21:47 (-0500) skreiv Jeff Garzik:

Quoted 5 lines
Quoted 3 lines

Does running an 'ls' on the server in the exported directory that is stale on the client resolve the problem (temporarily)?

Yes.

This still looks very much like a server issue to me. Could someone who is seeing the bug try to capture an instance of the ESTALE error going across the wire, and then do a fresh lookup of the same file from an "ls" call. I'd like to check how the stale filehandle differs from the freshly looked up one...

Please use "tcpdump -s 9000 -w /tmp/binary.pcap port 2049 and host my.servers.name" for the actual capture.

Cheers,
Trond

Signature · 1 line
Trond Myklebust <[email protected]>
#6 lore raw
Quoted 8 lines · Trond Myklebust

to den 04.11.2004 Klokka 21:47 (-0500) skreiv Jeff Garzik:

Quoted 4 lines
Quoted 2 lines

Does running an 'ls' on the server in the exported directory that is stale on the client resolve the problem (temporarily)?

Yes.

Quoted 8 lines

This still looks very much like a server issue to me. Could someone who is seeing the bug try to capture an instance of the ESTALE error going across the wire, and then do a fresh lookup of the same file from an "ls" call. I'd like to check how the stale filehandle differs from the freshly looked up one...

Please use "tcpdump -s 9000 -w /tmp/binary.pcap port 2049 and host my.servers.name" for the actual capture.

Will do.

FWIW my server is running 2.6.9-final. Client mount options in fstab are "defaults,tcp" and server options are (rw,no_root_squash,async).

Jeff

#7 lore raw
In reply to Jeff Garzik
Quoted 1 line · Jeff Garzik

Jakob Oestergaard wrote:

...

Quoted 8 lines
Quoted 5 lines

Hi Jeff,

Does running an 'ls' on the server in the exported directory that is stale on the client resolve the problem (temporarily)?

Yes.

Hah! Is that weird, or is that weird? :)

Probably whoever wants to look into this might want to get in touch with Anders as well:
http://lkml.org/lkml/2004/11/2/107

A quick workaround is to have cron run an 'ls' for you in the exported directory on the server side - b-e-a-utiful! :)

Signature · 1 line
 / jakob
#8 lore raw
In reply to Jeff Garzik
Quoted 10 lines · Jeff Garzik

I'm also seeing stale filehandle problems here in recent kernels.

Setup: x86 or x86-64, TCP, NFSv4 compiled in to both server and client, but not specified in mount options.

This is readily reproducible with rsync -- I just boot to an earlier version of the kernel on the NFS client, and the stale filehandle problems go away.

JEff

I am seeing severe nfs problems between kernels again, 2.4 - > 2.6 -> 2.4 V3 and V4 problems connecting and filesize mismatches between actual/reported. Might point to where the bug is.

Jeff

#9 lore raw

ty den 02.11.2004 Klokka 14:05 (-0700) skreiv Jeff V. Merkey:

Quoted 4 lines
Quoted 1 line

I am seeing severe nfs problems between kernels again, 2.4 - > 2.6 -> 2.4 V3 and V4 problems connecting and filesize mismatches between actual/reported. Might point to where the bug is.

Nope. I'm not seeing that at all (besides, that is entirely unrelated to ESTALE errors).

Mind telling us how to reproduce the problem?

Cheers,
Trond

Signature · 1 line
Trond Myklebust <[email protected]>

A message that is not in the archive (sent off-list or not yet received): [email protected]

#11 lore raw
Quoted 35 lines · [email protected] wrote:
Quoted 12 lines

Nope. I'm not seeing that at all (besides, that is entirely unrelated to ESTALE errors).

Mind telling us how to reproduce the problem?

Cheers,
Trond

Signature · 1 line
Trond Myklebust <[email protected]>

Connect 2.4.18 and 2.6.9 with NFS 3 enabled. I am seeing problems connecting and file size mismatches. I also see errors with zero length files (host side) that get opened and populated with data
and the remote side is unable to read them -- keeps seeing them as zero length.

I can setup this back up tommorrow morning in the lab and provide you all sorts of good debug and trace info. The problem seems to happen if the local side from the FS has open handles and starts writing blocks to a zero length file, and the remote side doesn't seem to see the data right away.

On typical FS read behavior, even if the file has a zero length, this
seems to be ignored and read requests continue until read returns
0 bytes read for file size. On the remote side, if the file size is not reflected, this is not the behavior, and probably should be. I can see someone truncating or changing a file size and NFS gets into a weird state.

Mismatches between configured v2/v3/v4 NFS configs between kernels
also have some issues. Please provide me with a sample /etc/exports
config you wish me to use exporting a directory you think would be helpful and I'll test and provide pcap traces of the traffic between the nodes.

Jeff

#12 lore raw

ty den 02.11.2004 Klokka 17:19 (-0700) skreiv Jeff V. Merkey:

Quoted 5 lines
Quoted 5 lines

Connect 2.4.18 and 2.6.9 with NFS 3 enabled. I am seeing problems connecting and file size mismatches. I also see errors with zero length files (host side) that get opened and populated with data and the remote side is unable to read them -- keeps seeing them as zero length.

That's entirely expected. NFS has always been forced to use a polling model for attribute cache consistency. "man 5 nfs" and read all about the "actimeo" mount options that control this behaviour.

Cheers,
Trond

Signature · 1 line
Trond Myklebust <[email protected]>
#13 lore raw
Quoted 10 lines · Trond Myklebust

ty den 02.11.2004 Klokka 17:19 (-0700) skreiv Jeff V. Merkey:

Quoted 5 lines
Quoted 5 lines

Connect 2.4.18 and 2.6.9 with NFS 3 enabled. I am seeing problems connecting and file size mismatches. I also see errors with zero length files (host side) that get opened and populated with data
and the remote side is unable to read them -- keeps seeing them as zero length.

Quoted 9 lines

That's entirely expected. NFS has always been forced to use a polling model for attribute cache consistency. "man 5 nfs" and read all about the "actimeo" mount options that control this behaviour.

Cheers,
Trond

Trond,

Thanks for the update. I noticed from another post on this thread that the problems with /etc/exports are being addressed. This was the other problem I was seeing but it appears to be getting fixed.

Jeff

#14 lore raw
Quoted 17 lines · Jeff V. Merkey
Quoted 6 lines · Trond Myklebust
Quoted 5 lines
Quoted 4 lines

Connect 2.4.18 and 2.6.9 with NFS 3 enabled. I am seeing problems connecting and file size mismatches. I also see errors with zero length files (host side) that get opened and populated with data
and the remote side is unable to read them -- keeps seeing them as zero length.

Quoted 9 lines

That's entirely expected. NFS has always been forced to use a polling model for attribute cache consistency. "man 5 nfs" and read all about the "actimeo" mount options that control this behaviour.

Cheers,
Trond

Quoted 10 lines

Trond,

Thanks for the update. I noticed from another post on this thread that the problems with /etc/exports are being addressed. This was the other problem I was seeing but it appears to be getting fixed.

Jeff

Trond,

While you are running down this problem, I've located another odd behavior you may (or may not) chose to address. In my dsfs file system, I use sector runs (large ones) that can sometimes return odd sizes during running down a file for reading.

i.e. I can return 4096,4096,4096,512,1024,4096,4096 block sizes on size returns
from vfs_read() for 512 or 4096 block requests when the file is read sequentially. Under NTFS and other OS architectures, unless the final read attempt returns 0 size denoting end of file, this seems to be allowed. I noticed that all of the linux code with
the exception of NFS also handles this situation quite nicely. NFS does not. I have
noted that if NFS receives a read size smaller than the requested block, it always assumes
end of file and terminates the next read. I hace modified dsfs to always return
block sizes in a uniform manner so NFS will work properly, even though the rest
of the Linux apps work just fine without assuming the end of file has been reached.

I personally think this is a broken behavior, but perhaps it's in line with some NFS spec somewhere. I have coded around it, but thout I would mention it to you.

Jeff

#15 lore raw

on den 03.11.2004 Klokka 14:38 (-0700) skreiv Jeff V. Merkey:

Quoted 4 lines

I personally think this is a broken behavior, but perhaps it's in line with some NFS
spec somewhere. I have coded around it, but thout I would mention it to you.

This should already be fixed in 2.6.x kernels.

Cheers,
Trond

Signature · 1 line
Trond Myklebust <[email protected]>
#16 lore raw
Quoted 9 lines · Trond Myklebust

on den 03.11.2004 Klokka 14:38 (-0700) skreiv Jeff V. Merkey:

Quoted 5 lines

I personally think this is a broken behavior, but perhaps it's in line with some NFS spec somewhere. I have coded around it, but thout I would mention it to you.

Quoted 7 lines

This should already be fixed in 2.6.x kernels.

Cheers,
Trond

I'll test it and let you know. I saw this on a 2.4.18 kernel in a 2.6.X NFS network test setup. I have not verified it on 2.6.X. Thanks for the heads up, sounds like you fixed it.

Jeff

#17 lore raw
In reply to Jeff Garzik

ty den 02.11.2004 Klokka 14:49 (-0500) skreiv Jeff Garzik:

Quoted 3 lines

This is readily reproducible with rsync -- I just boot to an earlier version of the kernel on the NFS client, and the stale filehandle problems go away.

Huh? The client cannot generate stale filehandle errors: only the server does that.
Have you got a binary tcpdump that shows the problem?

Cheers
Trond

Signature · 1 line
Trond Myklebust <[email protected]>
#18 lore raw
Quoted 8 lines · Trond Myklebust

ty den 02.11.2004 Klokka 14:49 (-0500) skreiv Jeff Garzik:

Quoted 1 line

This is readily reproducible with rsync -- I just boot to an earlier version of the kernel on the NFS client, and the stale filehandle problems go away.

Huh? The client cannot generate stale filehandle errors: only the server does that.

Not saying that the client is _generating_ the stale filehandle errors, only saying that they appear to go away when I boot the _client_ into older 2.6.9 kernels.

Quoted 1 line

Have you got a binary tcpdump that shows the problem?

I'll create one if I get time and can reliably reproduce it (rsync _sometimes_ shows the behavior, but not always).

Jeff

#19 lore raw

to den 04.11.2004 Klokka 21:57 (-0500) skreiv Jeff Garzik:

Quoted 3 lines

Not saying that the client is _generating_ the stale filehandle errors, only saying that they appear to go away when I boot the _client_ into older 2.6.9 kernels.

That would point to some pretty nasty memory corruption issues on the client then (affecting the cached filehandle in the inode itself).

So... I can't see that any NFS client changes have been pushed to Linus after the release of 2.6.9-rc2. Is the latter afflicted with the ESTALE problem?

Cheers,
Trond

Signature · 1 line
Trond Myklebust <[email protected]>
#20 lore raw
Quoted 6 lines · Trond Myklebust

to den 04.11.2004 Klokka 21:57 (-0500) skreiv Jeff Garzik:

Quoted 1 line

Not saying that the client is _generating_ the stale filehandle errors, only saying that they appear to go away when I boot the _client_ into older 2.6.9 kernels.

Quoted 6 lines

That would point to some pretty nasty memory corruption issues on the client then (affecting the cached filehandle in the inode itself).

So... I can't see that any NFS client changes have been pushed to Linus after the release of 2.6.9-rc2. Is the latter afflicted with the ESTALE problem?

I'll give it a test and find out...

Jeff

#21 lore raw
In reply to Jeff Garzik
Quoted 10 lines · Jeff Garzik
Quoted 10 lines

ty den 02.11.2004 Klokka 16:02 (+0400) skreiv Brad Campbell:

Quoted 7 lines

/raid 192.168.2.81(rw,async,no_root_squash)
/raid 192.168.3.80(rw,async,no_root_squash)
/raid0 192.168.2.81(rw,async,no_root_squash)
/raid0/tmp 192.168.2.81(rw,async,no_root_squash)
/raid2 192.168.2.81(rw,async,no_root_squash)
/raid2 192.168.3.80(rw,async,no_root_squash)
/nfsroot 192.168.2.81(rw,async,no_root_squash)

....

Quoted 8 lines

I'm also seeing stale filehandle problems here in recent kernels.

Setup: x86 or x86-64, TCP, NFSv4 compiled in to both server and client, but not specified in mount options.

This is readily reproducible with rsync -- I just boot to an earlier version of the kernel on the NFS client, and the stale filehandle problems go away.

Are any of the people seeing these problems able to reproduce them with the no_subtree_check export option set?

--b.

#22 lore raw
Quoted 6 lines · J. Bruce Fields
Quoted 5 lines · Jeff Garzik

I'm also seeing stale filehandle problems here in recent kernels.

Setup: x86 or x86-64, TCP, NFSv4 compiled in to both server and client, but not specified in mount options.

Quoted 6 lines
Quoted 3 lines

This is readily reproducible with rsync -- I just boot to an earlier version of the kernel on the NFS client, and the stale filehandle problems go away.

Are any of the people seeing these problems able to reproduce them with the no_subtree_check export option set?

I've finally managed to reproduce it here. It looks to me like it's failing in nfsd_acceptable, so exporting with no_subtree_check (which should probably be the default anyway) should eliminate the stale filehandle errors. Still trying to figure out why this is happening, though.

--Bruce Fields

#23 lore raw
In reply to Trond Myklebust
Quoted 11 lines · Trond Myklebust

ty den 02.11.2004 Klokka 16:02 (+0400) skreiv Brad Campbell:

Quoted 7 lines

/raid 192.168.2.81(rw,async,no_root_squash)
/raid 192.168.3.80(rw,async,no_root_squash)
/raid0 192.168.2.81(rw,async,no_root_squash)
/raid0/tmp 192.168.2.81(rw,async,no_root_squash)
/raid2 192.168.2.81(rw,async,no_root_squash)
/raid2 192.168.3.80(rw,async,no_root_squash)
/nfsroot 192.168.2.81(rw,async,no_root_squash)

You should only have 1 line per directory.

But exportfs won't complain about this.

Additionally, it seems to be allowed to use NIS netgroups in the exports file, except, well, it "sort of doesn't quite entirely always work". The best part of course being the errors you get: None on the server - but mounts might misbehave randomly at some point in the future on some client machines.

I just edited an exports file (according to your recommendations - because my server occationally gave stale file handles on some client machines), and found that the netmask 255.255.254.0 doesn't work either
- you need to specify two nets with 255.255.255.0 instead. My point
being, that even though I administer a network that I would describe as "small", even I run into major problems with the lack of proper parsing and error reporting of /etc/exports.

Generally, it seems exportfs would accept virtually any input, export it one way or another to the kernel, with unpredictable results and spurious errors at random times in the future.

What I would like was for exportfs to either say "No! Fix your file stupid" or "Good! This setup will work reliably for all eternity then".

Is anyone considering fixing this? And; is the problem mainly in
exportfs, in the kernel, or both? (relevant for me to know if I want to
go fix it myself) Thanks!

Signature · 1 line
 / jakob
#24 lore raw
Quoted 6 lines · Jakob Oestergaard

What I would like was for exportfs to either say "No! Fix your file stupid" or "Good! This setup will work reliably for all eternity then".

Is anyone considering fixing this? And; is the problem mainly in
exportfs, in the kernel, or both? (relevant for me to know if I want to go fix it myself)

With recent kernels the kernel generally only queries userland for export information on demand. So if you want an error returned at exportfs time, you need to modify exportfs.--b.

#25 lore raw
In reply to Jakob Oestergaard

on den 03.11.2004 Klokka 16:10 (+0100) skreiv Jakob Oestergaard:

Quoted 33 lines · Trond Myklebust
Quoted 11 lines

ty den 02.11.2004 Klokka 16:02 (+0400) skreiv Brad Campbell:

Quoted 7 lines

/raid 192.168.2.81(rw,async,no_root_squash)
/raid 192.168.3.80(rw,async,no_root_squash)
/raid0 192.168.2.81(rw,async,no_root_squash)
/raid0/tmp 192.168.2.81(rw,async,no_root_squash)
/raid2 192.168.2.81(rw,async,no_root_squash)
/raid2 192.168.3.80(rw,async,no_root_squash)
/nfsroot 192.168.2.81(rw,async,no_root_squash)

You should only have 1 line per directory.

But exportfs won't complain about this.

Additionally, it seems to be allowed to use NIS netgroups in the exports file, except, well, it "sort of doesn't quite entirely always work". The best part of course being the errors you get: None on the server - but mounts might misbehave randomly at some point in the future on some client machines.

I just edited an exports file (according to your recommendations - because my server occationally gave stale file handles on some client machines), and found that the netmask 255.255.254.0 doesn't work either
- you need to specify two nets with 255.255.255.0 instead. My point
being, that even though I administer a network that I would describe as "small", even I run into major problems with the lack of proper parsing and error reporting of /etc/exports.

Generally, it seems exportfs would accept virtually any input, export it one way or another to the kernel, with unpredictable results and spurious errors at random times in the future.

Quoted 6 lines

What I would like was for exportfs to either say "No! Fix your file stupid" or "Good! This setup will work reliably for all eternity then".

Is anyone considering fixing this? And; is the problem mainly in
exportfs, in the kernel, or both? (relevant for me to know if I want to
go fix it myself) Thanks!

I agree entirely.

The above two questions should really be directed to Neil (so I've Cced him). My guess would be that he would welcome any help here.

Although the limitations on what you are allowed to export are kernel-based, I believe you only need to add the checks to the nfs-utils package's parser for /etc/exports.

Cheers,
Trond

Signature · 1 line
Trond Myklebust <[email protected]>