I/O Errors with hard mounts
Subsystems (2)
Does=A0/var/log/messages show any errors around the same time?=A0 In ad= dition=A0to the=A0network trace=A0and=A0rpcdebug on the client, take a = look at "nfsstat -d" on the filer.=A0Is=A0the filer=A0dropping the conn= ection?=A0 Look for "dropped with EAGAIN" or "dropped from vol offline"=
in the output.=A0 This will help narrow down the problem.
- ricardo
Quoted 19 lines
-----Original Message-----
From: David Konerding [mailto:[email protected]]=20
Sent: Wednesday, June 04, 2008 6:33 AM
To: [email protected]
Subject: [NFS] I/O Errors with hard mounts
=20
Hi,
=20
We have a bunch of Linux clients (SLES 10 SP1) which mount a=20 NetApp filer.
=20
When the NetApp gets very, very busy, for example, one user is deleting 1Tbyte of data
while another user is doing a 30 client throughput test, it will stop responding to some requests.
=20
Although we are using hard mounts, some users report that during the hammering period, some of their
file operations produce "I/O Error" messages on their terminal.
Quoted 8 lines
=20
We checked, and the hosts are indeed using hard mounting.=A0 From our reading, I/O Errors
should only ever make it back to the user if are using soft mounting. =20
We're pretty sure the filer is not sending back an NFS_ERR=20 response (and we're
pretty sure that wouldn't get reported to the user as an I/O Error...=
)
Quoted 8 lines
=20
At this point, we suspect there must be a path in the NFS implementation that returns I/O Error to user space even with a hard mount.
=20
Any ideas?
=20
Dave
=20
-----------------------------------------------------------------------=
--
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued. Please subscribe to [email protected] instead.
http://vger.kernel.org/vger-lists.html#linux-nfs
A message that is not in the archive (sent off-list or not yet received): 927260.87785.qm-KtJlQ5K7SlOvuULXzWHTWIglqE1Y4D90QQ4Iyu8u01E@public.gmane.org
Quoted 1 line · Ricardo Labiaga
Does
/var/log/messagesshow any errors around the same time? In addition to the network trace and rpcdebug on the client, take a look at "nfsstat -d" on the filer. Is the filer dropping the connection? Look for "dropped with EAGAIN" or "dropped from vol offline" in the output. This will help narrow down the problem.
So, sometimes when somebody deletes a lot of data (like the problem we just observed),
the deleting host, and often other hosts, do report 'filer not responding' in the logs.
However, operations that aren't happening in the delete dir, tend to work just fine (for example, iozone could be running and doing pretty well)). Further, the most recent time this happened, the host didn't report filer not responding.
This is the only EAGAN reference I see:
assist queue (queued, split mbufs, drop for EAGAIN) = (0, 64478612, 94340)
Dave
-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that [email protected] is being discontinued. Please subscribe to [email protected] instead.
http://vger.kernel.org/vger-lists.html#linux-nfs