Skip to contentSkip to conversation

kswapd using a lot of CPU

DISCUSSION Steffen Persvold 2 participants 2 messages Nov 15, 2001

lore
Share & tools
lore
#0 lore raw

Dear kernel/VM gurus,

What the f**k is going on here (RedHat 7.1 kernel-2.4.9-12smp, glibc-2.2.4-19) ?

10:58pm up 6 days, 12:06, 1 user, load average: 3.33, 3.31, 3.20
146 processes: 142 sleeping, 4 running, 0 zombie, 0 stopped CPU0 states: 36.2% user, 63.3% system, 35.4% nice, 0.0% idle CPU1 states: 84.1% user, 15.1% system, 84.2% nice, 0.0% idle
Mem: 512244K av, 202012K used, 310232K free, 1072K shrd, 13480K buff
Swap: 2048248K av, 0K used, 2048248K free 58708K cached

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND

    5 root      20   0     0    0     0 RW   78.4  0.0  97:45 kswapd
 3342 magma     20   2 71200  43M 15196 R N  60.2  8.7  54:20 MAGMAfill
 3341 magma     18   2 71516  44M 16044 R N  60.0  8.8  54:37 MAGMAfill
# vmstat 2
   procs                      memory    swap          io     system         cpu
 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us  sy  id
 2  0  1      0 309100  13480  58932   2   3    10    15   31    32  18   1   4
 2  0  1      0 309076  13480  58940   0   0     0    28  180   161  60  40   0
 2  0  1      0 309020  13480  58944   0   0     0    46  215   191  60  40   0
 2  0  1      0 309080  13480  58948   0   0     0    20  152   143  61  39   0
 2  0  1      0 309088  13480  58948   0   0     0     6  131   141  59  41   0
 2  0  1      0 309064  13480  58952   0   0     0    12  109   129  59  41   0
 2  0  1      0 309028  13480  58956   0   0     0    56  167   153  57  43   0
 2  0  1      0 308912  13480  58960   0   0     0    32  211   201  60  39   0
 2  0  1      0 309072  13480  58960   0   0     0     0  115   118  62  38   0
 2  0  1      0 309072  13480  58960   0   0     0    12  134   142  60  40   0
 2  0  1      0 309076  13480  58960   0   0     0     0  104   122  59  41   0
 2  0  1      0 309072  13480  58960   0   0     0     0  130   127  61  39   0

kswapd is using plenty of CPU (this is, as you can see, a dual processor box, more exactly 2xPIII 1GHz), but there's no swapping going on ?!?. And even more strange is that there is actually nothing that is swapped out (2GB free).

If I try the following :

# cat /proc/swaps 
Filename                        Type            Size    Used    Priority
/dev/hda7                       partition       2048248 0       -1
# /sbin/swapoff /dev/hda7

Still kswapd is using a lot of CPU :

11:07pm up 6 days, 12:15, 1 user, load average: 3.33, 3.33, 3.24
146 processes: 141 sleeping, 5 running, 0 zombie, 0 stopped

CPU0 states: 40.0% user, 58.0% system, 40.0% nice,  0.1% idle
CPU1 states: 69.1% user, 30.0% system, 68.0% nice,  0.0% idle
Mem:   512244K av,  324476K used,  187768K free,    1072K shrd,   13608K buff
Swap:       0K av,       0K used,       0K free                  151296K cached

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND

    5 root      20   0     0    0     0 RW   80.0  0.0 105:19 kswapd
 3342 magma     20   2 65036  37M 15368 R N  62.3  7.5  60:10 MAGMAfill
 3341 magma     18   2  104M  78M 16096 R N  60.5 15.7  60:12 MAGMAfill

When I do :

# /sbin/swapon /dev/hda7

Nothing changes for kswapd, but the swap device gets a lower priority :

# cat /proc/swaps 
Filename                        Type            Size    Used    Priority
/dev/hda7                       partition       2048248 0       -2

If I repetedly do 'swapoff/swapon' the Priority decreases each time.

If I try a test application I've written which just mallocs a given amount of memory (I tried 450MBytes in this case), memsets it and then frees it, I'm able to force something to swap out, but after the application has exited kswapd is still using a lot of CPU.

The only thing that seems to solve the issue is to kill the two user processes (MPI processes) that is also using CPU.

Any feedback is highly appreciated,

Thanks,

Signature · 4 lines
  Steffen Persvold   | Scalable Linux Systems |   Try out the world's best   
 mailto:[email protected]  |  http://www.scali.com  | performing MPI implementation:
Tel: (+47) 2262 8950 |   Olaf Helsets vei 6   |      - ScaMPI 1.12.2 -         
Fax: (+47) 2262 8951 |   N0621 Oslo, NORWAY   | >300MBytes/s and <4uS latency
#1 lore raw
Quoted 5 lines · Steffen Persvold

What the f**k is going on here (RedHat 7.1 kernel-2.4.9-12smp, glibc-2.2.4-19) ?

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND

    5 root      20   0     0    0     0 RW   78.4  0.0  97:45 kswapd

Upgrade to a newer kernel, 2.4.10 and onward should solve this problem. The newest kernel is 2.4.14 -- just go for that.

Robert Love