Skip to contentSkip to conversation

Performance/stability problems with nfs shares

DISCUSSION J. Bruce Fields 1 participant 1 message Aug 5, 2013 1 message not in the archive

Subsystems (2)
lore
Share & tools
lore
#1 lore raw
In reply to a message not in the archive
Quoted 18 lines · Dawid Stawiarski

Dnia Piątek, 2 Sierpnia 2013 17:47 J. Bruce Fields napisał(a)

Quoted 8 lines · Dawid Stawiarski
Quoted 7 lines

W dniu 02.08.2013 15:12, Jeff Layton pisze:

Quoted 5 lines

Typically, a stack trace like that indicates that the process is waiting for the server to respond. The first thing I would do would be to ascertain whether the server is actually responding to these requests.

Quoted 8 lines
Quoted 2 lines

The same share is accessible on other nodes, so the problem involves only one of the nodes (completly random) at a time.

It's still conceivable that a server problem could cause it to stop responding to calls only from a single client--it'd be useful if possible to check a trace to see if that's what's happening. If the traffic is really huge then capturing and analyzing a good trace may be difficult.

Quoted 6 lines

I've managed to capture network traffic on client side, when the problem starts. It seems like:
0. operation SETATTR before problem works as charm (attached for reference)
1. retransmission happens very fast after sending 3 packets for WRITE operation 1a. linux is not using jumbo frame (3 packets of ~1K size instead of one)
2. linux ignores the ACK received after retransmission (actually it's SACK for the third packet)

I don't see any SACK?

But yes I see it appears to be ACKing a sequence number that should cover the segment the client keeps retransmitting.

And I don't know what would cause that.

Might be worth asking networking folks? ([email protected]).

--b.

Quoted 5 lines

3. both client and server start to retransmit their packets

what happens next (not visible in the pcap attached) is that the server sends RST after some time, and client starts new connection - after that, things start to work normally again. So maybe, it's more general problem with TCP stack and not NFS itself?

D.