Skip to contentSkip to conversation

NFS client problems in 2.4.18 to 2.4.20

DISCUSSION Joshua Weage 4 participants 13 messages Sep 5, 2003 – Sep 10, 2003

lore
Share & tools
lore
#0 lore raw

I hope this was not discussed previously, I couldn't find anything relevant in the archives.

I am having problems with NFS clients getting stuck after reporting a "nfs server not responding message". The majority of the time the mount starts working again when the nfs server load goes down. However, sometimes the mount on one client becomes completely unresponsive, but all of the clients still work correctly. Even after letting it set for 2-3+ hours it still doesn't come back up. I can ping the server from the locked client and that works. If I do a lazy unmount and then remount the NFS disk it works again for awhile - but tends to lock up again. A standard umount doesn't work when the client is hung.

This happens with all RedHat kernel releases 2.4.18 to 2.4.20.

I have tried tuning the NFS server by going to nfs utils 1.0.3 and by increasing nfsd's and the socket buffer sizes. I have also increased the timeout on the clients to 2.0. One thing that seems to help is to enable async mode on the NFS server. However, I've still seen the same client hang with async turned on.

Machine Details:
12x Cluster nodes 2xAMD Athlon MP's, 100 MbEthernet 1x server 2xPentium III 1.13GHz, Adaptec 39160, Promise RM8000, GigEthernet
1x Cisco 2924-T switch.

I'm running 8 CPU jobs, each cpu occasionally writes 120MB files to the NFS disk. The client lockup always occurs during these file writes. The lockups have occured on several of the cluster nodes.

Any suggestions on what could be causing this?

Thanks,

Joshua Weage

=====

__________________________________
Do you Yahoo!?
Yahoo! SiteBuilder - Free, easy-to-use web site design software
http://sitebuilder.yahoo.com

#1 lore raw
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

" " == Joshua Weage <[email protected]> writes:

> Any suggestions on what could be causing this?

Have you read through the sections pertaining to these problems in the NFS HOWTO and NFS FAQ? If not, see http://nfs.sourceforge.net

Cheers,
Trond

#2 lore raw

I have read through them and did some tuning a few months ago. The only thing that seems relevant to this problem is UDP buffer overflows, but I'm using the 2.4.20 kernel. I'm using 8192 byte read and write sizes and have increased the number of nfsd's and socket buffer size on the server. Everything was working fine until about 4 weeks ago when the mounts on the clients started locking up. The server is dropping packets at times, but increasing nfds only manages to load the disk further - it doesn't seem to reduce timeouts. However, I don't understand why a mount on a single client will go AWOL - and never come back - while all the others will continue to work properly.

Are there any commands that would allow me to figure out why the mount has stopped working? I've looked at nfsstat and the kernel seems to have stopped sending any data to the server, or it may send one packet every couple of seconds. If I start up another shell and try to do an ls on the problem filesystem, the command locks up and can't be interrupted. I think I've also mounted the same filesystem in another location, on the same machine, and it works fine.

Josh

Quoted 10 lines · --- Trond Myklebust
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

" " == Joshua Weage <[email protected]> writes:

> Any suggestions on what could be causing this?

Have you read through the sections pertaining to these problems in the
NFS HOWTO and NFS FAQ? If not, see http://nfs.sourceforge.net

Cheers,
Trond

=====

__________________________________
Do you Yahoo!?
Yahoo! SiteBuilder - Free, easy-to-use web site design software
http://sitebuilder.yahoo.com

#3 lore raw
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

" " == Joshua Weage <[email protected]> writes:

> Are there any commands that would allow me to figure out why > the mount has stopped working? I've looked at nfsstat and the > kernel seems to have stopped sending any data to the server, or > it may send one packet every couple of seconds. If I start up > another shell and try to do an ls on the problem filesystem, > the command locks up and can't be interrupted. I think I've > also mounted the same filesystem in another location, on the > same machine, and it works fine.

Does 'dmesg' produce any clues as to what is going on?

How about tcpdump?

Cheers,
Trond

#4 lore raw

There aren't any clues in the kernel logs, except that the kernel does report "nfs server not responding" and never comes back with "nfs server OK". I've enabled kernel debugging on all of the cluster nodes, but the above message is all that I get in the logs.

I'll have to try out tcpdump the next time this happens.

Thanks,

Joshua Weage

Quoted 5 lines

Does 'dmesg' produce any clues as to what is going on?

How about tcpdump?

=====

__________________________________
Do you Yahoo!?
Yahoo! SiteBuilder - Free, easy-to-use web site design software
http://sitebuilder.yahoo.com

#5 lore raw
Quoted 6 lines · Joshua Weage

There aren't any clues in the kernel logs, except that the kernel does report "nfs server not responding" and never comes back with "nfs server OK". I've enabled kernel debugging on all of the cluster nodes, but the above message is all that I get in the logs.

I'll have to try out tcpdump the next time this happens.

Look for lots of retransmits from the client. This might be the bug where it adjusts the retransmit timeout to a ridiculously small sub-millisecond value, because of a sequence of fast cached responses from the server, then when the server responds slowly due to a disk access the client times out within milliseconds. Repeatedly.

-- Jamie

#6 lore raw
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

" " == Jamie Lokier <[email protected]> writes:

> Look for lots of retransmits from the client. This might be > the bug where it adjusts the retransmit timeout to a > ridiculously small sub-millisecond value, because of a sequence > of fast cached responses from the server, then when the server > responds slowly due to a disk access the client times out > within milliseconds. Repeatedly.

Nope. He said 2.4.18 to 2.4.20...

Cheers,
Trond

#7 lore raw
In reply to Jamie Lokier
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

" " == Jamie Lokier <[email protected]> writes:

> This might be the bug where it adjusts the retransmit timeout > to a ridiculously small sub-millisecond value, because of a > sequence of fast cached responses from the server

BTW: this should be fixed now in 2.6.x. I've set a minimum value on the estimated error on the round-trip time to 1/10sec.

Cheers,
Trond

#8 lore raw
Quoted 6 lines · Trond Myklebust

> This might be the bug where it adjusts the retransmit timeout > to a ridiculously small sub-millisecond value, because of a > sequence of fast cached responses from the server

BTW: this should be fixed now in 2.6.x. I've set a minimum value on the estimated error on the round-trip time to 1/10sec.

I don't think the round-trip estimate was ever a real problem, although setting a low bound does make sense.

A real problem is the rule of having a fixed number of retransmits before an operation fails with a "soft" moount. This is wrong for NFS, now that rtt is estimated dynamically.

It is wrong because NFS response times are not due to network congestion - they are due mainly to I/O on the server, and I/O times don't have the same properties as networks at all.

The "soft operation fail" imeout should have a minimum absolute time, like 30 seconds or so. It should also have a maximum (for systems where the estimated rtt is 10 seconds). This should be independent of the rtt estimate.

Think of a worst case:

- server responds to cached requests within 10 microseconds

- uncached requests take 10 seconds to respond (spinning up CD, seeking on tape HFS, or just ordinary disk/swap contention).

This should never timeout, as long as the server is responding within a fixed absolute time, although it's fine to issue lots of retransmits until that time.

The fundamental error is assuming that all NFS requests take about the same time to server, and delays are caused by the network. This isn't true especially on a LAN. Delays for NFS are typically caused by I/O, and vary by 6 orders of magnitude from request to request.

-- Jamie

#9 lore raw
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

" " == Jamie Lokier <[email protected]> writes:

> A real problem is the rule of having a fixed number of > retransmits before an operation fails with a "soft" moount. > This is wrong for NFS, now that rtt is estimated dynamically.

> The "soft operation fail" imeout should have a minimum absolute > time, like 30 seconds or so. It should also have a maximum > (for systems where the estimated rtt is 10 seconds). This > should be independent of the rtt estimate.

> - server responds to cached requests within 10 microseconds

> - uncached requests take 10 seconds to respond (spinning up CD,
> seeking on tape HFS, or just ordinary disk/swap
> contention).

This is not an issue for tapes, etc. NFS has an alternative mechanisms for dealing with this in the form of the NFSERR_JUKEBOX error.

However for disks I agree that you do have 'large' variations between the cached and uncached case. Should latency really be much larger than 1/10 second for a 32k read though?

> The fundamental error is assuming that all NFS requests take > about the same time to server, and delays are caused by the > network. This isn't true especially on a LAN. Delays for NFS > are typically caused by I/O, and vary by 6 orders of magnitude > from request to request.

If it was merely a case of random error, then we wouldn't have a problem at all. The RTT code does make an estimate of the error on the the measurement. The problem is that there is a large tail in the graph of round trip time vs. number of events due to these disk spinups, etc...

However retransmissions compensate somewhat because they impose a geometric increase in the timeout value. i.e. The for the first transmission the timeout == the rto, then the retransmissions follow 2*rto, 4*rto, 8*rto,...

Part of the problem in the Linux case is therefore that we have a too low default value for 'retrans'. The kernel default is '5' (same as for Solaris, however the mount program still overrides that default with a value of '3'. This implies that for soft mounts, we never wait longer than 15*rto before we time out (well - 7*rto actually since the code in xdr_adjust_timeout() actually appears to confuse number of retransmissions with the number of transmissions).

By setting 'retrans=6' (5 + 1 to compensate for the bug), therefore, people can ensure that we retry for at least 6 seconds before timing out. The question is: is this an adequate default?

Cheers,
Trond

#10 lore raw
Quoted 2 lines · Trond Myklebust

This is not an issue for tapes, etc. NFS has an alternative mechanisms for dealing with this in the form of the NFSERR_JUKEBOX error.

Oh, cool. Perhaps the server should send these automatically, when I/O operations are taking a little bit too long?

Quoted 3 lines

However for disks I agree that you do have 'large' variations between the cached and uncached case. Should latency really be much larger than 1/10 second for a 32k read though?

Yes, it often is. I seek 32k reads taking several seconds, when the disk is busy with other tasks.

Quoted 11 lines

> The fundamental error is assuming that all NFS requests take > about the same time to server, and delays are caused by the > network. This isn't true especially on a LAN. Delays for NFS > are typically caused by I/O, and vary by 6 orders of magnitude > from request to request.

If it was merely a case of random error, then we wouldn't have a problem at all. The RTT code does make an estimate of the error on the the measurement. The problem is that there is a large tail in the graph of round trip time vs. number of events due to these disk spinups, etc...

Yes, exactly. Though I saw the tail due to normal disk seek + access time is quite significant, compared with cached access time.

Quoted 13 lines

However retransmissions compensate somewhat because they impose a geometric increase in the timeout value. i.e. The for the first transmission the timeout == the rto, then the retransmissions follow 2*rto, 4*rto, 8*rto,...

Part of the problem in the Linux case is therefore that we have a too low default value for 'retrans'. The kernel default is '5' (same as for Solaris, however the mount program still overrides that default with a value of '3'. This implies that for soft mounts, we never wait longer than 15*rto before we time out (well - 7*rto actually since the code in xdr_adjust_timeout() actually appears to confuse number of retransmissions with the number of transmissions).

Quoted 3 lines

By setting 'retrans=6' (5 + 1 to compensate for the bug), therefore, people can ensure that we retry for at least 6 seconds before timing out. The question is: is this an adequate default?

That would be a big improvement. I take it you have effectively clamped the retransmit time at a minimum of 1/10 second, then? (I didn't understand what you meant earlier).

Last time I used a soft mount, I was seeing the first retransmit after some time smaller than a millisecond. (I don't remember, but 0.1ms sounds about right). If that is the retransmit time, then retrans=6 won't be enough - retrans=16 would be needed. I don't think a good correct retrans=xxx setting should depend on the network like that. Setting a minimum retransmit time is one way to fix that.

-- Jamie

#11 lore raw
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line

" " == Jamie Lokier <[email protected]> writes:

This is not an issue for tapes, etc. NFS has an alternative mechanisms for dealing with this in the form of the NFSERR_JUKEBOX error.

> Oh, cool. Perhaps the server should send these automatically, > when I/O operations are taking a little bit too long?

Yes. Needs a patch to knfsd, but would be very useful for people that want to export tapes, CD exchangers, etc...

Quoted 4 lines
Quoted 4 lines

By setting 'retrans=6' (5 + 1 to compensate for the bug), therefore, people can ensure that we retry for at least 6 seconds before timing out. The question is: is this an adequate default?

> That would be a big improvement. I take it you have > effectively clamped the retransmit time at a minimum of 1/10 > second, then? (I didn't understand what you meant earlier).

Yes. When we calculate the timeout value, we add the estimated error*4 to the estimated round trip time. I've set a floor on the former value so that the minumum timeout will be 1/10second.

> Last time I used a soft mount, I was seeing the first > retransmit after some time smaller than a millisecond. (I > don't remember, but 0.1ms sounds about right). If that is the > retransmit time, then retrans=6 won't be enough - retrans=16 > would be needed. I don't think a good correct retrans=xxx > setting should depend on the network like that. Setting a > minimum retransmit time is one way to fix that.

It is already in 2.6.0. I'm expecting to put it into 2.4.23 too, but I want to know that this (together with a patch to 'mount' to change the retrans default) really does solve the problem for people...

Cheers,
Trond

#12 lore raw
In reply to Joshua Weage
Quoted 8 lines · Joshua Weage

has stopped working? I've looked at nfsstat and the kernel seems to have stopped sending any data to the server, or it may send one packet every couple of seconds. If I start up another shell and try to do an ls on the problem filesystem, the command locks up and can't be interrupted. I think I've also mounted the same filesystem in another location, on the same machine, and it works fine.

I am experiencing similar problems with 2.4.18 as a client (the NFS server is on Solaris). When the client freezes I see nfsstat 'client rpc retrans' counting fast. I found a quite strange way to unlock the machine, by performing a port scan with nmap from elsewhere. BTW, which Gigabit ethernet do you use, I use tg3, there could be a relation to the network card?

Cheers,
Wouter