Skip to contentSkip to conversation

Writing over NFS causes lots of paging

DISCUSSION Simon Kirby 3 participants 3 messages Nov 11, 2001 – Nov 12, 2001

lore
Share & tools
lore
#0 lore raw

It looks like when writing large amounts of data to NFS where the remote end is slower than the local end the local end appears to start swapping out a lot I'm guessing this is because it can read much faster than it can write.

Also, I see NFS timeouts and thus "I/O error" messages fom cp when it is mounted with the "soft" option, even with high timeouts. "hard" works fine, but I didn't want to use it for this mount.

For example:

procs memory swap io system cpu

r b w   swpd  free buff  cache si   so   bi   bo   in    cs us sy id
0 0 0  70316  4128  608 141744  0    0 4172    0 3065  4214  1 12 87
0 0 0  70316  3156  620 142704  0   92 3896   92 2904  4028  1 11 88
0 1 0  70316  4208  672 141604  0   40 2912   40 3248  4715  2 12 86
1 0 0  70316  3228  724 142528  0  296 3284  296 2904  4682  5 17 79
0 1 0  70700  4048  720 141948  0  124 3744  124 2987  4109  1 14 85
1 1 0  70700  3752  724 142376  0  116 4136  116 2927  3985  1 16 83
1 0 0  70956  3816  712 142384  0  180 3964  180 2724  3801  1 15 84
0 0 0  70956  3968  720 142308  0    0 3908    0 3045  4277  2 13 84
0 1 0  71724  3336  724 142984  0  580 3796  580 2837  4985  9 21 69
0 0 0  71724  3924  736 144116  0  588 3776  588 2860  3950  1 14 85
0 1 0  72236  3120  752 146132  0  556 3380  556 2731  3983  1  9 89
0 0 0  73260  3212  752 146140  0 2496 3468 2496 2516  3637  1 14 85
0 0 0  73900  3476  744 145868  0  640 3900  640 2776  3888  0 13 87
0 0 0  74156  3192  736 146212  0  540 4150  540 2916  4010  1 15 83

The copy is still running and almost everything is swapped out now (140 MB). When the copy started, there was about 30 MB of swap.

NFS client (reading from disk and writing through NFS): 2.4.15pre1 NFS server (writing to disk from NFS): 2.4.15pre2
NFSv3 and knfsd used.

Is there something different with the VM here? Should I try 2.4.15pre2 on the NFS client?

Simon-

[ Stormix Technologies Inc. ][ NetNation Communications Inc. ]
[ [email protected] ][ [email protected] ]
[ Opinions expressed are not necessarily those of my employers. ]

#1 lore raw

In article <[email protected]> you write:

Quoted 4 lines

It looks like when writing large amounts of data to NFS where the remote end is slower than the local end the local end appears to start swapping out a lot I'm guessing this is because it can read much faster than it can write.

No, the real reason for why the NFS write stuff causes page-outs is that the VM layer does not really understand the notion of writeback pages.

The VM layer has one explicit special case: it knows about the magic in "page->buffers", and can handle writeback for block-oriented devices sanely. But any non-buffer-oriented filesystem is "invisible" to the VM layer, and has to use other tricks to make the VM ignore its pages.

In the case of NFS, it increments the page count and has it's own private non-VM-visible writeback data structures. This pins the page in memory, but at the same time, because the VM doesn't understand it, the VM will end up thinking the page is mapped in user space or something else, and won't know how to start writeouts.

Quite frankly, I don't rightly know what the real fix is. Making "page->buffers" be a generic thing (a "void *") along with making the buffer flushing logic be behind a address space operation is probably the right thing in the long run.

Linus

#2 lore raw
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

" " == Linus Torvalds <[email protected]> writes:

> The VM layer has one explicit special case: it knows about the > magic in "page->buffers", and can handle writeback for > block-oriented devices sanely. But any non-buffer-oriented > filesystem is "invisible" to the VM layer, and has to use other > tricks to make the VM ignore its pages.

<snip>

> Quite frankly, I don't rightly know what the real fix > is. Making "page->buffers" be a generic thing (a "void *") > along with making the buffer flushing logic be behind a address > space operation is probably the right thing in the long run.

That only takes care of writebacks. Don't forget that reading & readahead can also eat memory if somebody forgets to call lock_page() (a common problem on 'hard,intr' mounts).

You'll notice that in the NFS updates I sent you the other day, there is a new function 'nfs_try_to_free_pages()' that provides a rather generic way of freeing up NFS memory resources. Its sole purpose today is to ensure that we keep an upper limit of 256 requests per mount.

My (still somewhat vague) plan is to expand that interface some time during 2.5.x to allow the VM to control and limit the memory usage of the NFS client - flushing out read and write requests if necessary. IMHO, the filesystem can often be more efficient at clearing out pages if we leave the choice of strategy up to it, rather than having the VM micro-manage exactly which page is to be thrown out first. For instance, under NFSv3 there is usually a huge advantage to sending off a COMMIT over any other call, since it can potentially free up a whole truckload of pages.

Cheers,
Trond