nfsd and kswapd
Subsystems (2)
Hi,
I'm observing a strange phenomenon running nfsd on 2.4.20 (RH). I'm = using a
dual-Xeon server with 6GB of memory, and 40 nfsd threads running. Under very heavy load, after a while kswapd starts constantly occupying = between
60-90% cpu, and every once in a while keventd also wakes up with some = 10-
30% of its own. This obviously slows the system and greatly reduces the = nfs
performance. The strangest thing is, that when I reduce the memory to = 512MB
(with the mem boot option), kswapd seems to be getting a lot less CPU.
Any explanations on the matter would be greatly appreciated.
Thanks,
Amir.
Here's a dump of top and /proc/meminfo:
08:22:17 up 3:23, 3 users, load average: 41.85, 41.38, 38.05 102
processes: 98 sleeping, 4 running, 0 zombie, 0 stopped
CPU0 states: 0.0% user 98.5% system 0.0% nice 0.0% iowait 1.1%
idle
CPU1 states: 0.3% user 98.4% system 0.0% nice 0.0% iowait 0.6%
idle
CPU2 states: 0.0% user 100.0% system 0.0% nice 0.0% iowait 0.0%
idle
CPU3 states: 0.0% user 99.0% system 0.0% nice 0.0% iowait 0.6%
idle
Mem: 6178484k av, 6161280k used, 17204k free, 0k shrd, 106908k
buff
1896336k actv, 1881924k in_d, 17864k in_c
Swap: 1028120k av, 9756k used, 1018364k free 5544656k
cached
PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU =
COMMAND
11 root 25 0 0 0 0 DW 76.0 0.0 77:20 3 kswapd
24902 root 16 0 1184 1160 856 R 15.4 0.0 2:17 1 top
6 root 15 0 0 0 0 SW 14.8 0.0 12:29 0 =
keventd
22616 root 15 0 0 0 0 DW 11.3 0.0 1:31 1 nfsd
22603 root 16 0 0 0 0 DW 10.7 0.0 1:32 1 nfsd
22626 root 15 0 0 0 0 DW 10.3 0.0 1:29 3 nfsd
22598 root 16 0 0 0 0 DW 10.2 0.0 1:30 1 nfsd
22620 root 15 0 0 0 0 DW 10.0 0.0 1:29 3 nfsd
22590 root 16 0 0 0 0 DW 9.6 0.0 1:30 0 nfsd
22608 root 15 0 0 0 0 DW 9.6 0.0 1:31 3 nfsd
22596 root 15 0 0 0 0 DW 9.5 0.0 1:32 3 nfsd
22621 root 16 0 0 0 0 DW 9.5 0.0 1:32 0 nfsd
22617 root 15 0 0 0 0 DW 9.0 0.0 1:33 0 nfsd
22591 root 15 0 0 0 0 DW 8.9 0.0 1:34 3 nfsd
22605 root 15 0 0 0 0 DW 8.9 0.0 1:31 2 nfsd
22611 root 15 0 0 0 0 DW 8.6 0.0 1:31 1 nfsd
22614 root 15 0 0 0 0 DW 8.3 0.0 1:35 3 nfsd
22612 root 15 0 0 0 0 DW 8.0 0.0 1:33 2 nfsd
22625 root 16 0 0 0 0 RW 8.0 0.0 1:32 1 nfsd
22601 root 16 0 0 0 0 DW 7.9 0.0 1:34 1 nfsd
22610 root 15 0 0 0 0 DW 7.9 0.0 1:32 3 nfsd
22613 root 15 0 0 0 0 DW 7.8 0.0 1:33 1 nfsd
22600 root 15 0 0 0 0 DW 7.5 0.0 1:31 2 nfsd
22604 root 15 0 0 0 0 DW 7.3 0.0 1:32 1 nfsd
22624 root 15 0 0 0 0 DW 7.3 0.0 1:35 1 nfsd
22594 root 15 0 0 0 0 RW 7.0 0.0 1:32 3 nfsd
22595 root 15 0 0 0 0 DW 7.0 0.0 1:32 1 nfsd
22587 root 16 0 0 0 0 DW 6.9 0.0 1:36 0 nfsd
22599 root 15 0 0 0 0 DW 6.8 0.0 1:33 2 nfsd
22588 root 16 0 0 0 0 DW 6.3 0.0 1:34 3 nfsd
22622 root 16 0 0 0 0 DW 6.2 0.0 1:36 0 nfsd
22592 root 15 0 0 0 0 DW 5.8 0.0 1:32 2 nfsd
22609 root 15 0 0 0 0 DW 5.8 0.0 1:35 3 nfsd
22597 root 15 0 0 0 0 DW 5.5 0.0 1:31 3 nfsd
22619 root 15 0 0 0 0 DW 5.3 0.0 1:33 0 nfsd
22593 root 15 0 0 0 0 RW 5.2 0.0 1:33 2 nfsd
22615 root 15 0 0 0 0 DW 5.2 0.0 1:29 3 nfsd
22606 root 15 0 0 0 0 DW 4.5 0.0 1:32 1 nfsd
22623 root 15 0 0 0 0 DW 4.5 0.0 1:32 0 nfsd
22589 root 15 0 0 0 0 DW 4.3 0.0 1:33 3 nfsd
22618 root 15 0 0 0 0 DW 4.2 0.0 1:32 0 nfsd
22607 root 15 0 0 0 0 DW 4.1 0.0 1:35 0 nfsd
22602 root 16 0 0 0 0 DW 3.9 0.0 1:31 1 nfsd
24936 nir 15 0 7612 7072 5088 S 2.6 0.1 0:07 1
gnome-terminal
13 root 15 0 0 0 0 SW 0.1 0.0 0:08 0
kscand/Normal
789 root 15 0 940 484 364 S 0.1 0.0 0:43 0 nmbd
1 root 15 0 108 80 56 S 0.0 0.0 0:04 0 init[root@BigMama root]# cat /proc/meminfo
total: used: free: shared: buffers: cached:
Mem: 6326767616 6309052416 17715200 0 107671552 5681623040 Swap: 1052794880 9883648 1042911232 MemTotal: 6178484 kB MemFree: 17300 kB MemShared: 0 kB Buffers: 105148 kB Cached: 5545584 kB SwapCached: 2876 kB Active: 1904520 kB ActiveAnon: 6456 kB ActiveCache: 1898064 kB Inact_dirty: 1876544 kB Inact_laundry: 1862944 kB Inact_clean: 13176 kB Inact_target: 1131436 kB HighTotal: 5373824 kB HighFree: 1024 kB LowTotal: 804660 kB LowFree: 16276 kB SwapTotal: 1028120 kB SwapFree: 1018468 kB
-------------------------------------------------------
This SF.Net email is sponsored by the new InstallShield X.
Hi Amir,
Quoted 10 lines · Amir Hermelin
Hi,
I'm observing a strange phenomenon running nfsd on 2.4.20 (RH). I'm using a dual-Xeon server with 6GB of memory, and 40 nfsd threads running. Under very heavy load, after a while kswapd starts constantly occupying between 60-90% cpu, and every once in a while keventd also wakes up with some 10- 30% of its own. This obviously slows the system and greatly reduces the nfs performance. The strangest thing is, that when I reduce the memory to 512MB (with the mem boot option), kswapd seems to be getting a lot less CPU.Any explanations on the matter would be greatly appreciated.
Is that a RH 7.3 system? If so, make sure you're running kernel 2.4.20-28 or later, it has the memory management of RH 9 backported and behaves much better in a setup like yours.
There were many discussions about this on the Dell Linux PowerEdge list, one thread started here:
http://lists.us.dell.com/pipermail/linux-poweredge/2003-May/024785.html
One workaround is to disable swap, not sure if that's on option for you though. Depends on what you're running other than file serving.
Greetings,
Marc
-------------------------------------------------------
This SF.Net email is sponsored by the new InstallShield X.
Thanks Marc for the info. It so happens I'm also using a PowerEdge = (2600),
so I'll check out that thread. I'm using a RH 9 system, kernel 2.4.20-8 = -
should that include the better mem management you've mentioned?
-----Original Message-----
From: Marc Schmitt [mailto:[email protected]]=20
Sent: Monday, June 07, 2004 10:22 AM
To: Amir Hermelin
Cc: [email protected]
Subject: Re: [NFS] nfsd and kswapd
Hi Amir,
Quoted 11 lines · Amir Hermelin
Hi,
I'm observing a strange phenomenon running nfsd on 2.4.20 (RH). I'm=20 using a dual-Xeon server with 6GB of memory, and 40 nfsd threads=20 running. Under very heavy load, after a while kswapd starts=20 constantly occupying between 60-90% cpu, and every once in a while=20 keventd also wakes up with some 10- 30% of its own. This obviously=20 slows the system and greatly reduces the nfs performance. The=20 strangest thing is, that when I reduce the memory to 512MB (with the=20 mem boot option), kswapd seems to be getting a lot less CPU. =20
Any explanations on the matter would be greatly appreciated.
Is that a RH 7.3 system? If so, make sure you're running kernel = 2.4.20-28 or
later, it has the memory management of RH 9 backported and behaves much better in a setup like yours.
There were many discussions about this on the Dell Linux PowerEdge list, = one
thread started here:
http://lists.us.dell.com/pipermail/linux-poweredge/2003-May/024785.html
One workaround is to disable swap, not sure if that's on option for you though. Depends on what you're running other than file serving.
Greetings,
Marc
-------------------------------------------------------
This SF.Net email is sponsored by the new InstallShield X.
Hi Amir,
Quoted 5 lines · Amir Hermelin
Thanks Marc for the info. It so happens I'm also using a PowerEdge (2600), so I'll check out that thread. I'm using a RH 9 system, kernel 2.4.20-8 - should that include the better mem management you've mentioned?
Yes, but 2.4.20-8 is very old and does not have the kswapd fix. Try upgrading to 2.4.20-31 (https://rhn.redhat.com/errata/RHSA-2004-166.html), it should make a difference.
Greetings,
Marc
-------------------------------------------------------
This SF.Net email is sponsored by the new InstallShield X.