Problem: Clients freeze on transfer of large files, w gigabit lan
Subsystems (2)
Good Day All,
I and many others have been plagued by a problem that can be summarised
as follows:
Client hangs upon copying of large files TO the server. Transfer begins quickly then hangs, sometimes taking the client o/s with it, until transfer starts again. In less extreme cases transfer is sporadic.
I am using nfs4 w gigabit nics.
My experience is with debian and ubuntu clients and servers, as such I
started a thread on ubuntu forum to find similar cases to give you more to work from:
http://ubuntuforums.org/showthread.php?p=9269703
It appears that this problem is not restricted to ubuntu and exists in
at
least fedora and redhat as well.
More details and user experience is in the above forum thread.
Help in solving this would be greatly appreciated.
Thanks.
Jasper
Quoted 17 lines · Jasper Mackenzie
Good Day All,
I and many others have been plagued by a problem that can be summarised
as follows:Client hangs upon copying of large files TO the server. Transfer begins quickly then hangs, sometimes taking the client o/s with it, until transfer starts again. In less extreme cases transfer is sporadic.
I am using nfs4 w gigabit nics.My experience is with debian and ubuntu clients and servers, as such I
started a thread on ubuntu forum to find similar cases to give you more to work from:
http://ubuntuforums.org/showthread.php?p=9269703
It appears that this problem is not restricted to ubuntu and exists in
at
least fedora and redhat as well.
Quoted 3 lines
More details and user experience is in the above forum thread.
Help in solving this would be greatly appreciated.
Could this perhaps be related to the following bugzilla entry?
https://bugzilla.kernel.org/show_bug.cgi?id=16213
If so, then could you please try the proposed fix and see if it helps.
Cheers
Trond
Quoted 12 lines
Quoted 11 lines
I and many others have been plagued by a problem that can be
summarised
as follows:Client hangs upon copying of large files TO the server. Transfer begins quickly then hangs, sometimes taking the client o/s with it, until transfer starts again. In less extreme cases transfer is sporadic.
I am using nfs4 w gigabit nics.
http://ubuntuforums.org/showthread.php?p=9269703
It appears that this problem is not restricted to ubuntu and exists
Quoted 7 lines
Could this perhaps be related to the following bugzilla entry?
https://bugzilla.kernel.org/show_bug.cgi?id=16213If so, then could you please try the proposed fix and see if it helps.
Cheers
Trond
Thanks Trond,
The patch solved the client lockups with a patched vanilla kernel (will keep trying with an ubuntu kernel, as it should do the same, but didnt)
Unfortunatley it dousnt fix the other problem, as it seems they are separate, of the transfer happening in bursts, reducing the actual throughput dramatically. i.e transfer starts at 16mb/s (according to nautilus), then 3 or 4 seconds later the progress bar stops, the hdd activity also stops (its the same with cp of course, nautilus just gives me a good indication of xfer speed), then 4 or 5 seconds later it starts again for 3 or 4 seconds.... repeat ad nausium...
Refer to the forum thread for graphs etc. of throughput if nesc.
Thanks.
Jasper
Quoted 22 lines · Jasper Mackenzie
Quoted 11 lines
Quoted 10 lines
I and many others have been plagued by a problem that can be
summarised
as follows:Client hangs upon copying of large files TO the server. Transfer begins quickly then hangs, sometimes taking the client o/s with it, until transfer starts again. In less extreme cases transfer is sporadic.
I am using nfs4 w gigabit nics.
http://ubuntuforums.org/showthread.php?p=9269703
It appears that this problem is not restricted to ubuntu and existsQuoted 7 lines
Could this perhaps be related to the following bugzilla entry?
https://bugzilla.kernel.org/show_bug.cgi?id=16213If so, then could you please try the proposed fix and see if it helps.
Cheers
TrondThanks Trond,
The patch solved the client lockups with a patched vanilla kernel (will keep trying with an ubuntu kernel, as it should do the same, but didnt)
Quoted 8 lines
Unfortunatley it dousnt fix the other problem, as it seems they are separate, of the transfer happening in bursts, reducing the actual throughput dramatically. i.e transfer starts at 16mb/s (according to nautilus), then 3 or 4 seconds later the progress bar stops, the hdd activity also stops (its the same with cp of course, nautilus just gives me a good indication of xfer speed), then 4 or 5 seconds later it starts again for 3 or 4 seconds.... repeat ad nausium...
Refer to the forum thread for graphs etc. of throughput if nesc.
That is usually because the server is caching too much data instead of progressively writing it out. When the client calls 'commit' (the NFS equivalent of fsync()) then the disk on the server goes into a frenzy of writing, and the client does the RPC equivalent of twiddling its thumbs until the server is done...
I'd suggest trying to lower the values
of /proc/sys/vm/dirty_expire_centisecs
and /proc/sys/vm/dirty_background_ratio on the server. You might also try lowering /proc/sys/vm/dirty_writeback_centisecs...
Cheers
Trond
Quoted 40 lines
Quoted 25 lines
Quoted 6 lines
Quoted 6 lines
I and many others have been plagued by a problem that can be
summarised
as follows:Client hangs upon copying of large files TO the server. Transfer
begins
Quoted 5 lines
Quoted 5 lines
quickly then hangs, sometimes taking the client o/s with it, until transfer starts again. In less extreme cases transfer is sporadic.
I am using nfs4 w gigabit nics.
http://ubuntuforums.org/showthread.php?p=9269703
It appears that this problem is not restricted to ubuntu andexists
Quoted 8 lines
Could this perhaps be related to the following bugzilla entry?
https://bugzilla.kernel.org/show_bug.cgi?id=16213If so, then could you please try the proposed fix and see if it helps.
Cheers
TrondThanks Trond,
The patch solved the client lockups with a patched vanilla kernel (will keep trying with an ubuntu kernel, as it should do the same, but didnt)Quoted 8 lines
Unfortunatley it dousnt fix the other problem, as it seems they are separate, of the transfer happening in bursts, reducing the actual throughput dramatically. i.e transfer starts at 16mb/s (according to nautilus), then 3 or 4 seconds later the progress bar stops, the hdd activity also stops (its the same with cp of course, nautilus just gives me a good indication of xfer speed), then 4 or 5 seconds later it starts again for 3 or 4 seconds.... repeat ad nausium...
Refer to the forum thread for graphs etc. of throughput if nesc.That is usually because the server is caching too much data instead of progressively writing it out. When the client calls 'commit' (the NFS equivalent of
fsync()) then the disk on the server goes into a frenzy of writing, and the client does the RPC equivalent of twiddling its thumbs until the server is done...
Quoted 7 lines
I'd suggest trying to lower the values
of/proc/sys/vm/dirty_expire_centisecs
and/proc/sys/vm/dirty_background_ratioon the server. You might also try lowering/proc/sys/vm/dirty_writeback_centisecs...Cheers
Trond
I reduced them to 10% of the value I found them at to 40,1,20
respectivly, with no improvement.
I think I need to play with it more to see if the lockups are gone. Ime not sure if the original problem is quite fixed... dammit. was too easy !
When tested with ubuntu more, I will see how the ppl on the above forum
go.
any other ideas?
Thanks
Jasper