nfs performance at high loads
Hello,
I had sent in a note on nfs performance issues some time back,
and Mark Hemment had been kind enough to point out to the zerocopy networking patch. Well, we tried with it, and it does seem to have some improvement, but it seems to have screwed up nfs performance a bit, because we see a LOT of rpc failures for all kinds of calls, starting from lookup, to read and writes. Could this possibly be triggered by this patch ( picked up from davem's site for 2.4.0 ).
On the other hand, we do plan to migrate to 2.4.2. Can somebody
update me or provide pointers to info. as to whether we can expect some of these problems have been resolved in 2.4.2? We should soon be testing on 2.4.2
Thanks
________________________________________________
Get your own "800" number
Voicemail, fax, email, and a lot more
http://www.ureach.com/reg/tag
---- On Wed, 4 Apr 2001, Mark Hemment ([email protected])
Quoted 2 lines · wrote:
I believe David Miller's latest zero-copy patches might help
here.
Quoted 1 line
In his patch, the pull-up buffer is now allocated near the
top of
Quoted 3 lines
stack
(in the sunrpc code), so it can be a blocking allocation.
This doesn't fix the core VM problems, but does relieve the
pressure
Quoted 1 line
_slightly_ on the VM (I assume, haven't tried David's patch
yet).
Quoted 3 lines
One of the core problems is that the VM keeps no measure of
page fragmentation in the free page pool. The system reaches
the state
Quoted 2 lines
of
having plenty of free single pages (so kswapd and friends
aren't kicked
Quoted 1 line
- or if they are, they do no or little word), and very few
buddied pages
Quoted 3 lines
(which you need for some of the NFS requests).
Unfortunately, even with keeping a mesaure of fragmentation,
and
Quoted 1 line
insuring work is done when it is reached, doesn't solve the
next
Quoted 3 lines
problem.
When a large order request comes in, the
inactive_cleanpage
list is
Quoted 1 line
reaped. As
reclaim_page()simply selects the "oldest" page it
can, with
Quoted 1 line
no regard as to whether it will buddy (now, or 'possibily in
the near
Quoted 1 line
future), this list is quickly shrunk by a large order request
- far too
Quoted 3 lines
quickly for a well behaved system.
An NFS write request, with an 8K block size, needs an
order-2 (16K)
Quoted 2 lines
pull
up buffer (we shouldn't really be pulling the header into the
same
Quoted 2 lines
buffer
as the data - perhaps we aren't any more?). On a well used
system, an
Quoted 1 line
order-2 _blocking_ allocation ends up populating the order-0
and order-1
Quoted 5 lines
with quite a few pages from the
inactive_clean.This then triggers another problem. :(
As large (non-zero) order requests are always from the
NORMAL or DMA
Quoted 1 line
zones, these zones tend to have a lot of free-pages (put there
by the
Quoted 1 line
blind
reclaim_page()- well, once you can do a blocking
allocation they
Quoted 2 lines
are, or when the fragmentation kicking is working).
New allocations for pages for the page-cache often ignore
the HIGHMEM
Quoted 1 line
zone (it reaches a steady state), and so is passed over by the
loop at
Quoted 3 lines
the
head of__alloc_pages()).
However, NORMAL and DMA zones tend to be abovepages_low
(due to the
Quoted 1 line
reason above), and so new page-cache pages came from these
zones. On a
Quoted 1 line
HIGHMEM system this leads to thrashing of the NORMAL zone,
while the
Quoted 2 lines
HIGHMEM zone stays (relatively) quiet.
Note: To make matters even worse under this condition,
pulling pages
Quoted 2 lines
out
of the NORMAL zone is exactly what you don't want to happen!
It would
Quoted 2 lines
be
much better if they could be left alone for a (short) while to
give them
Quoted 1 line
chance to buddy - Linux (at present) doesn't care about the
budding of
Quoted 1 line
pages in the HIGHMEM zone (no non-zero allocations come from
there).
Quoted 2 lines
I was working on these problems (and some others) a few
months back,
Quoted 2 lines
and
will to return to them shortly. Unfortunately, the changes
started to
Quoted 2 lines
look too large for 2.4....
Also, for NFS, the best solution now might be to give the
nfsd threads
Quoted 2 lines
a
receive buffer. With David's patches, the pull-up occurs in
the context
Quoted 2 lines
of a thread, making this possible.
This doesn't solve the problem for other subsystems which do
non-zero
Quoted 1 line
order page allocations, but (perhaps) they have a low enough
frequency
Quoted 7 lines
not
to be of real issue.Kapish,
Note: Ensure you put a "sync" in your
/etc/exports- the
default
Quoted 8 lines
behaviour was "async" (not legal for a valid SpecFS run).
Mark
Quoted 1 line · Alan Cox
Quoted 1 line
We have been seeing some problems with running nfs
benchmarks
Quoted 1 line
Quoted 1 line
Quoted 1 line
at very high loads and were wondering if somebody could
show
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
some pointers to where the problem lies.
The system is a 2.4.0 kernel on a 6.2 Red at distribution
( so
Quoted 2 lines
Quoted 2 lines
Use 2.2.19. The 2.4 VM is currently too broken to survive
high I/O
Quoted 8 lines
benchmark
Quoted 4 lines
tests without going silly
-
To unsubscribe from this list: send the line "unsubscribelinux-kernel" in
Quoted 2 lines
the body of a message to [email protected] More majordomo info at
http://vger.kernel.org/majordomo-info.html
Quoted 9 lines
Quoted 2 lines
Please read the FAQ at http://www.tux.org/lkml/
-
To unsubscribe from this list: send the line "unsubscribe
linux-kernel"
Quoted 3 lines
in
the body of a message to [email protected] More majordomo info at
http://vger.kernel.org/majordomo-info.html
Quoted 3 lines
Please read the FAQ at http://www.tux.org/lkml/
Quoted 16 lines · Kapish K
Hello,
I had sent in a note on nfs performance issues some time back,
and Mark Hemment had been kind enough to point out to the zerocopy networking patch. Well, we tried with it, and it does seem to have some improvement, but it seems to have screwed up nfs performance a bit, because we see a LOT of rpc failures for all kinds of calls, starting from lookup, to read and writes. Could this possibly be triggered by this patch ( picked up from davem's site for 2.4.0 ).
On the other hand, we do plan to migrate to 2.4.2. Can somebody
update me or provide pointers to info. as to whether we can expect some of these problems have been resolved in 2.4.2? We should soon be testing on 2.4.2
Thanks
While I'm not an expert hacker or anything, I can tell you for sure, that even 2.4.2 is full of really system crippling bugs. You need to track the current kernels. All of the 2.4.x series should be compatible, in other words, you should upgrade as soon as possible to the latest stable kernel. Currently, that's 2.4.3 (not 2.4.2). And even 2.4.3 has many known bugs that are capable of 1) destroying performance and 2) destroying filesystems.
At the very least, you should upgrade to 2.4.3, but better yet would be to upgrade to the 2.4.4 when it comes out (soon?), or even the 2.4.4-pre patch since it has the zero copy networking patch already included, as well as fixes for bugs that could corrupt your filesystems. The zero copy patch as it existed for 2.4.0 was also buggy in itself, so that would explain some of your extended problems.
Really, 2.4.0 is a 'horrible' kernel to be running, as it is missing an enormous amount of performance fixes, and bug fixes.
David
Signature · 3 lines
David Mansfield (718) 963-2020 [email protected] Ultramaster Group, LLC www.ultramaster.com