Skip to contentSkip to conversation

Problem: Clients freeze on transfer of large files, w gigabit lan

DISCUSSION Jasper Mackenzie 2 participants 5 messages Jun 25, 2010 – Jun 28, 2010 1 message not in the archive

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Good Day All,
I and many others have been plagued by a problem that can be summarised
as follows:

Client hangs upon copying of large files TO the server. Transfer begins quickly then hangs, sometimes taking the client o/s with it, until transfer starts again. In less extreme cases transfer is sporadic.
I am using nfs4 w gigabit nics.

My experience is with debian and ubuntu clients and servers, as such I
started a thread on ubuntu forum to find similar cases to give you more to work from:
http://ubuntuforums.org/showthread.php?p=9269703
It appears that this problem is not restricted to ubuntu and exists in
at
least fedora and redhat as well.

More details and user experience is in the above forum thread.

Help in solving this would be greatly appreciated.

Thanks.

Jasper

#1 lore raw
Quoted 17 lines · Jasper Mackenzie

Good Day All,
I and many others have been plagued by a problem that can be summarised
as follows:

Client hangs upon copying of large files TO the server. Transfer begins quickly then hangs, sometimes taking the client o/s with it, until transfer starts again. In less extreme cases transfer is sporadic.
I am using nfs4 w gigabit nics.

My experience is with debian and ubuntu clients and servers, as such I
started a thread on ubuntu forum to find similar cases to give you more to work from:
http://ubuntuforums.org/showthread.php?p=9269703
It appears that this problem is not restricted to ubuntu and exists in
at
least fedora and redhat as well.

Quoted 3 lines

More details and user experience is in the above forum thread.

Help in solving this would be greatly appreciated.

Could this perhaps be related to the following bugzilla entry?

    https://bugzilla.kernel.org/show_bug.cgi?id=16213

If so, then could you please try the proposed fix and see if it helps.

Cheers
Trond

#3 lore raw
In reply to a message not in the archive
Quoted 12 lines
Quoted 11 lines

I and many others have been plagued by a problem that can be
summarised
as follows:

Client hangs upon copying of large files TO the server. Transfer begins quickly then hangs, sometimes taking the client o/s with it, until transfer starts again. In less extreme cases transfer is sporadic.
I am using nfs4 w gigabit nics.
http://ubuntuforums.org/showthread.php?p=9269703
It appears that this problem is not restricted to ubuntu and exists

Quoted 7 lines

Could this perhaps be related to the following bugzilla entry?

    https://bugzilla.kernel.org/show_bug.cgi?id=16213

If so, then could you please try the proposed fix and see if it helps.

Cheers
Trond

Thanks Trond,
The patch solved the client lockups with a patched vanilla kernel (will keep trying with an ubuntu kernel, as it should do the same, but didnt)

Unfortunatley it dousnt fix the other problem, as it seems they are separate, of the transfer happening in bursts, reducing the actual throughput dramatically. i.e transfer starts at 16mb/s (according to nautilus), then 3 or 4 seconds later the progress bar stops, the hdd activity also stops (its the same with cp of course, nautilus just gives me a good indication of xfer speed), then 4 or 5 seconds later it starts again for 3 or 4 seconds.... repeat ad nausium...
Refer to the forum thread for graphs etc. of throughput if nesc.

Thanks.

Jasper

#4 lore raw
Quoted 22 lines · Jasper Mackenzie
Quoted 11 lines
Quoted 10 lines

I and many others have been plagued by a problem that can be
summarised
as follows:

Client hangs upon copying of large files TO the server. Transfer begins quickly then hangs, sometimes taking the client o/s with it, until transfer starts again. In less extreme cases transfer is sporadic.
I am using nfs4 w gigabit nics.
http://ubuntuforums.org/showthread.php?p=9269703
It appears that this problem is not restricted to ubuntu and exists

Quoted 7 lines

Could this perhaps be related to the following bugzilla entry?

    https://bugzilla.kernel.org/show_bug.cgi?id=16213

If so, then could you please try the proposed fix and see if it helps.

Cheers
Trond

Thanks Trond,
The patch solved the client lockups with a patched vanilla kernel (will keep trying with an ubuntu kernel, as it should do the same, but didnt)

Quoted 8 lines

Unfortunatley it dousnt fix the other problem, as it seems they are separate, of the transfer happening in bursts, reducing the actual throughput dramatically. i.e transfer starts at 16mb/s (according to nautilus), then 3 or 4 seconds later the progress bar stops, the hdd activity also stops (its the same with cp of course, nautilus just gives me a good indication of xfer speed), then 4 or 5 seconds later it starts again for 3 or 4 seconds.... repeat ad nausium...
Refer to the forum thread for graphs etc. of throughput if nesc.

That is usually because the server is caching too much data instead of progressively writing it out. When the client calls 'commit' (the NFS equivalent of fsync()) then the disk on the server goes into a frenzy of writing, and the client does the RPC equivalent of twiddling its thumbs until the server is done...

I'd suggest trying to lower the values
of /proc/sys/vm/dirty_expire_centisecs
and /proc/sys/vm/dirty_background_ratio on the server. You might also try lowering /proc/sys/vm/dirty_writeback_centisecs...

Cheers
Trond

#5 lore raw
Quoted 40 lines
Quoted 25 lines
Quoted 6 lines
Quoted 6 lines

I and many others have been plagued by a problem that can be
summarised
as follows:

Client hangs upon copying of large files TO the server. Transfer

begins

Quoted 5 lines
Quoted 5 lines

quickly then hangs, sometimes taking the client o/s with it, until transfer starts again. In less extreme cases transfer is sporadic.
I am using nfs4 w gigabit nics.
http://ubuntuforums.org/showthread.php?p=9269703
It appears that this problem is not restricted to ubuntu and

exists

Quoted 8 lines

Could this perhaps be related to the following bugzilla entry?

    https://bugzilla.kernel.org/show_bug.cgi?id=16213

If so, then could you please try the proposed fix and see if it helps.

Cheers
Trond

Thanks Trond,
The patch solved the client lockups with a patched vanilla kernel (will keep trying with an ubuntu kernel, as it should do the same, but didnt)

Quoted 8 lines

Unfortunatley it dousnt fix the other problem, as it seems they are separate, of the transfer happening in bursts, reducing the actual throughput dramatically. i.e transfer starts at 16mb/s (according to nautilus), then 3 or 4 seconds later the progress bar stops, the hdd activity also stops (its the same with cp of course, nautilus just gives me a good indication of xfer speed), then 4 or 5 seconds later it starts again for 3 or 4 seconds.... repeat ad nausium...
Refer to the forum thread for graphs etc. of throughput if nesc.

That is usually because the server is caching too much data instead of progressively writing it out. When the client calls 'commit' (the NFS equivalent of fsync()) then the disk on the server goes into a frenzy of writing, and the client does the RPC equivalent of twiddling its thumbs until the server is done...

Quoted 7 lines

I'd suggest trying to lower the values
of /proc/sys/vm/dirty_expire_centisecs
and /proc/sys/vm/dirty_background_ratio on the server. You might also try lowering /proc/sys/vm/dirty_writeback_centisecs...

Cheers
Trond

I reduced them to 10% of the value I found them at to 40,1,20
respectivly, with no improvement.
I think I need to play with it more to see if the lockups are gone. Ime not sure if the original problem is quite fixed... dammit. was too easy !

When tested with ubuntu more, I will see how the ppl on the above forum
go.

any other ideas?

Thanks

Jasper