nfs locking up system: rpc-srv/tcp: nfsd: got error -104

DISCUSSION Bob Kryger 2 participants 3 messages May 10, 2007

Subsystems (2)

NFS client NFS server

lore

Share & tools

lore

Bob Kryger

author

2007-05-10 15:28 UTC #0 lore raw

Can someone explain the error and possibly give me a hint as to how to move forward...

Thanks
Bob

Installed a new FC6 system:

[root@nfs3 log]# uname -a
Linux nfs3 2.6.18-1.2798.fc6 #1 SMP Mon Oct 16 14:37:32 EDT 2006 i686 i686 i386 GNU/Linux

The system is not stable - it locks up after the following message occurs (here are two occurances).

May  8 10:31:48 nfs3 kernel: rpc-srv/tcp: nfsd: got error -104 when sending 140 bytes - shutting down socket
May  9 18:20:32 nfs3 kernel: rpc-srv/tcp: nfsd: got error -104 when sending 132 bytes - shutting down socket

It seems to happen every day or so, but the system is not yet in production, so there is very little if and any traffic on it. I had originally though this had something to do with using rsize=8192,wsize=8192 on a client, while testing, but wonder if that may just have accelerated the issue, as it has now happened when there is a mount without such specs, and very little traffic (overnight).

There are no other error messages occurring with this one. (Is this even a NFS error, or possibly portmapper?)

Hardware:

BIOS Information

        Vendor: Phoenix Technologies LTD
        Version: 6.00
        Release Date: 10/30/2006

Base Board Information
Manufacturer: Supermicro
Product Name: PDSMi
Processor Information
Socket Designation: LGA775/U1
Type: Central Processor
Family: <OUT OF SPEC>
Manufacturer: Intel
ID: 49 0F 00 00 FF FB EB BF
Version: Intel(R) Celeron(R) D CPU 2.80GHz

e1000: eth1: e1000_probe: Intel(R) PRO/1000 Network Connection
e1000: eth0: e1000_watchdog: NIC Link is Up 1000 Mbps Full Duplex

QLogic Fibre Channel HBA Driver
ACPI: PCI Interrupt 0000:00:1f.3[B] -> GSI 19 (level, low) -> IRQ 233
ACPI: PCI Interrupt 0000:0a:01.0[A] -> GSI 24 (level, low) -> IRQ 58 qla2xxx 0000:0a:01.0: Found an ISP2300, irq 58, iobase 0xf8882000 qla2xxx 0000:0a:01.0: Configuring PCI space...
qla2xxx 0000:0a:01.0: Configure NVRAM parameters...
qla2xxx 0000:0a:01.0: Verifying loaded RISC code...
qla2xxx 0000:0a:01.0: Allocated (412 KB) for firmware dump... qla2xxx 0000:0a:01.0: Waiting for LIP to complete...
Floppy drive(s): fd0 is 1.44M
FDC 0 is a post-1991 82077
qla2xxx 0000:0a:01.0: LIP reset occured (f8f7).
qla2xxx 0000:0a:01.0: LIP occured (f8f7).
qla2xxx 0000:0a:01.0: LOOP UP detected (1 Gbps).
qla2xxx 0000:0a:01.0: Topology - (Loop), Host Loop address 0x8 scsi1 : qla2xxx
qla2xxx 0000:0a:01.0:
QLogic Fibre Channel HBA Driver: 8.01.07-k1
QLogic QLA2310 -
ISP2300: PCI-X (66 MHz) @ 0000:0a:01.0 hdma-, host#=1, fw=3.03.20 IPX

There are 8 disks like this one on the Qlogic which are set up in an md array and exported via NFS3

sd 1:0:0:0: Attached scsi disk sda
Vendor: SEAGATE Model: ST1181677FC Rev: 0001
Type: Direct-Access ANSI SCSI revision: 03

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

Jeff Layton

2007-05-10 15:43 UTC #1 lore raw

Quoted 8 lines · Bob Kryger

Can someone explain the error and possibly give me a hint as to how to move forward...
Thanks
Bob
Installed a new FC6 system:

Quoted 8 lines

[root@nfs3 log]# uname -a
Linux nfs3 2.6.18-1.2798.fc6 #1 SMP Mon Oct 16 14:37:32 EDT 2006 i686 i686 i386 GNU/Linux
The system is not stable - it locks up after the following message occurs (here are two occurances).
May  8 10:31:48 nfs3 kernel: rpc-srv/tcp: nfsd: got error -104 when sending 140 bytes - shutting down socket
May  9 18:20:32 nfs3 kernel: rpc-srv/tcp: nfsd: got error -104 when sending 132 bytes - shutting down socket

$ grep 104 /usr/include/asm-generic/errno*
/usr/include/asm-generic/errno.h:#define        ECONNRESET      104     /* Connection reset by peer */

I'd suspect some sort of network issue or a problem on the other end of the connection. Sniffing traffic on the wire while reproducing this might help to confirm it and determine where RST packets are coming from.

Quoted 62 lines

It seems to happen every day or so, but the system is not yet in production, so there is very little if and any traffic on it. I had originally though this had something to do with using rsize=8192,wsize=8192 on a client, while testing, but wonder if that may just have accelerated the issue, as it has now happened when there is a mount without such specs, and very little traffic (overnight).
There are no other error messages occurring with this one. (Is this even a NFS error, or possibly portmapper?)
Hardware:
BIOS Information
        Vendor: Phoenix Technologies LTD
        Version: 6.00
        Release Date: 10/30/2006
Base Board Information
Manufacturer: Supermicro
Product Name: PDSMi
Processor Information
Socket Designation: LGA775/U1
Type: Central Processor
Family: <OUT OF SPEC>
Manufacturer: Intel
ID: 49 0F 00 00 FF FB EB BF
Version: Intel(R) Celeron(R) D CPU 2.80GHz
e1000: eth1: e1000_probe: Intel(R) PRO/1000 Network Connection
e1000: eth0: e1000_watchdog: NIC Link is Up 1000 Mbps Full Duplex
QLogic Fibre Channel HBA Driver
ACPI: PCI Interrupt 0000:00:1f.3[B] -> GSI 19 (level, low) -> IRQ 233
ACPI: PCI Interrupt 0000:0a:01.0[A] -> GSI 24 (level, low) -> IRQ 58 qla2xxx 0000:0a:01.0: Found an ISP2300, irq 58, iobase 0xf8882000 qla2xxx 0000:0a:01.0: Configuring PCI space...
qla2xxx 0000:0a:01.0: Configure NVRAM parameters...
qla2xxx 0000:0a:01.0: Verifying loaded RISC code...
qla2xxx 0000:0a:01.0: Allocated (412 KB) for firmware dump... qla2xxx 0000:0a:01.0: Waiting for LIP to complete...
Floppy drive(s): fd0 is 1.44M
FDC 0 is a post-1991 82077
qla2xxx 0000:0a:01.0: LIP reset occured (f8f7).
qla2xxx 0000:0a:01.0: LIP occured (f8f7).
qla2xxx 0000:0a:01.0: LOOP UP detected (1 Gbps).
qla2xxx 0000:0a:01.0: Topology - (Loop), Host Loop address 0x8 scsi1 : qla2xxx
qla2xxx 0000:0a:01.0:
QLogic Fibre Channel HBA Driver: 8.01.07-k1
QLogic QLA2310 -
ISP2300: PCI-X (66 MHz) @ 0000:0a:01.0 hdma-, host#=1, fw=3.03.20 IPX
There are 8 disks like this one on the Qlogic which are set up in an md array and exported via NFS3
sd 1:0:0:0: Attached scsi disk sda
Vendor: SEAGATE Model: ST1181677FC Rev: 0001
Type: Direct-Access ANSI SCSI revision: 03

Quoted 9 lines

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

Jeff Layton

2007-05-10 15:46 UTC #2 lore raw

In reply to Bob Kryger

Quoted 6 lines · Bob Kryger

Can someone explain the error and possibly give me a hint as to how to move forward...
Thanks
Bob

Quoted 7 lines

Installed a new FC6 system:
[root@nfs3 log]# uname -a
Linux nfs3 2.6.18-1.2798.fc6 #1 SMP Mon Oct 16 14:37:32 EDT 2006 i686 i686 i386 GNU/Linux
The system is not stable - it locks up after the following message occurs (here are two occurances).

Also, that kernel is pretty old. You might consider patching to something more recent and seeing if that alleviates the problem...

Quoted 65 lines

May  8 10:31:48 nfs3 kernel: rpc-srv/tcp: nfsd: got error -104 when sending 140 bytes - shutting down socket
May  9 18:20:32 nfs3 kernel: rpc-srv/tcp: nfsd: got error -104 when sending 132 bytes - shutting down socket
It seems to happen every day or so, but the system is not yet in production, so there is very little if and any traffic on it. I had originally though this had something to do with using rsize=8192,wsize=8192 on a client, while testing, but wonder if that may just have accelerated the issue, as it has now happened when there is a mount without such specs, and very little traffic (overnight).
There are no other error messages occurring with this one. (Is this even a NFS error, or possibly portmapper?)
Hardware:
BIOS Information
        Vendor: Phoenix Technologies LTD
        Version: 6.00
        Release Date: 10/30/2006
Base Board Information
Manufacturer: Supermicro
Product Name: PDSMi
Processor Information
Socket Designation: LGA775/U1
Type: Central Processor
Family: <OUT OF SPEC>
Manufacturer: Intel
ID: 49 0F 00 00 FF FB EB BF
Version: Intel(R) Celeron(R) D CPU 2.80GHz
e1000: eth1: e1000_probe: Intel(R) PRO/1000 Network Connection
e1000: eth0: e1000_watchdog: NIC Link is Up 1000 Mbps Full Duplex
QLogic Fibre Channel HBA Driver
ACPI: PCI Interrupt 0000:00:1f.3[B] -> GSI 19 (level, low) -> IRQ 233
ACPI: PCI Interrupt 0000:0a:01.0[A] -> GSI 24 (level, low) -> IRQ 58 qla2xxx 0000:0a:01.0: Found an ISP2300, irq 58, iobase 0xf8882000 qla2xxx 0000:0a:01.0: Configuring PCI space...
qla2xxx 0000:0a:01.0: Configure NVRAM parameters...
qla2xxx 0000:0a:01.0: Verifying loaded RISC code...
qla2xxx 0000:0a:01.0: Allocated (412 KB) for firmware dump... qla2xxx 0000:0a:01.0: Waiting for LIP to complete...
Floppy drive(s): fd0 is 1.44M
FDC 0 is a post-1991 82077
qla2xxx 0000:0a:01.0: LIP reset occured (f8f7).
qla2xxx 0000:0a:01.0: LIP occured (f8f7).
qla2xxx 0000:0a:01.0: LOOP UP detected (1 Gbps).
qla2xxx 0000:0a:01.0: Topology - (Loop), Host Loop address 0x8 scsi1 : qla2xxx
qla2xxx 0000:0a:01.0:
QLogic Fibre Channel HBA Driver: 8.01.07-k1
QLogic QLA2310 -
ISP2300: PCI-X (66 MHz) @ 0000:0a:01.0 hdma-, host#=1, fw=3.03.20 IPX
There are 8 disks like this one on the Qlogic which are set up in an md array and exported via NFS3
sd 1:0:0:0: Attached scsi disk sda
Vendor: SEAGATE Model: ST1181677FC Rev: 0001
Type: Direct-Access ANSI SCSI revision: 03

Quoted 9 lines

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs