Skip to contentSkip to conversation

loadavg always equal or above 1.00 - how to explain?

DISCUSSION Tomasz Chmielewski 6 participants 10 messages Dec 1, 2005

lore
Share & tools
lore
#0 lore raw

I noticed one of my Samba + OpenLDAP servers, running 2.6.11.4 kernel has loadavg always equal or above 1.00, although I can't explain it.

# cat /proc/loadavg
1.00 1.10 1.06 1/65 782

This server is barely used, and as I remember, loadavg was always close to 0.00 on that system.

When I view the process list with top, no process takes more than 1% of CPU time; RAM usage is also minimal:

# free
             total       used       free     shared    buffers     cached
Mem:        320836     241016      79820          0      23308     177232
-/+ buffers/cache:      40476     280360
Swap:       811272      14612     796660

This has ~ 50 processes running (ps aux|wc -l), and ~ 50 network connections (netstat -tupna|wc -l), so everything normal.

Nothing unusual in dmesg, too.

What can cause this anormal load, and how can I spot it?

Signature · 3 lines
Tomek
http://wpkg.org
WPKG - software deployment and upgrades with Samba
#1 lore raw

Can you use top to determine which process is requesting most of the CPU?

Quoted 25 lines · Tomasz Chmielewski

I noticed one of my Samba + OpenLDAP servers, running 2.6.11.4 kernel has loadavg always equal or above 1.00, although I can't explain it.

# cat /proc/loadavg
1.00 1.10 1.06 1/65 782

This server is barely used, and as I remember, loadavg was always close to 0.00 on that system.

When I view the process list with top, no process takes more than 1% of CPU time; RAM usage is also minimal:

# free
              total       used       free     shared    buffers     cached
Mem:        320836     241016      79820          0      23308     177232
-/+ buffers/cache:      40476     280360
Swap:       811272      14612     796660

This has ~ 50 processes running (ps aux|wc -l), and ~ 50 network connections (netstat -tupna|wc -l), so everything normal.

Nothing unusual in dmesg, too.

What can cause this anormal load, and how can I spot it?

Signature · 2 lines
________
www.hipersonik.com : Open source experts
#3 lore raw
In reply to Tomasz Chmielewski
Quoted 2 lines · Tomasz Chmielewski

I noticed one of my Samba + OpenLDAP servers, running 2.6.11.4 kernel has loadavg always equal or above 1.00, although I can't explain it.

[...]

Quoted 1 line

What can cause this anormal load, and how can I spot it?

Some hidden rootkit ?

Xav

#5 lore raw
In reply to Tomasz Chmielewski
Quoted 6 lines · Tomasz Chmielewski

1.00 1.10 1.06 1/65 782

This server is barely used, and as I remember, loadavg was always close
to 0.00 on that system.

remember that load is the sum of running/runable processes and processes in D state (waiting for IO generally, but not always). I'm pretty sure your load comes from one of the later...

ps ought to tell you which one it is... (if not, an
"echo t > /proc/sysrq-trigger" will dump the kernel state including the offending process, and will also tell us where exactly that process is)

Greetings,
Arjan van de Ven

#4 lore raw
In reply to Xavier Bestel
Quoted 10 lines · Xavier Bestel
Quoted 1 line · Tomasz Chmielewski

I noticed one of my Samba + OpenLDAP servers, running 2.6.11.4 kernel has loadavg always equal or above 1.00, although I can't explain it.

[...]

Quoted 1 line

What can cause this anormal load, and how can I spot it?

Some hidden rootkit ?

highly unlikely, this machine is totally separated from the outer world, to break into it someone would have to break into several other servers first (and no problems with those).

Signature · 3 lines
Tomek
http://wpkg.org
WPKG - software deployment and upgrades with Samba
#2 lore raw
In reply to Norbert van Nobelen
Quoted 1 line · Norbert van Nobelen

Can you use top to determine which process is requesting most of the CPU?

Actually, when I press shift + P in top, top is the most used process for a while - around 1%, then it drops to ~ 0.0-0.3% and stays like that; other processes (like sshd, smbd) don't take more than ~0.5% really few times a minute.

Same goes with memory usage.

vmstat output:

procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us
sy id wa
2 0 14612 69748 26112 184004 0 0 1 1 3 3 3
3 94 0

iostat output:

avg-cpu:  %user   %nice    %sys %iowait   %idle
           1,79    1,52    2,66    0,33   93,70

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn

hda               2,51        49,99        50,09  538412978  539475864
hdb               0,00         0,00         0,00       1744          0
fd0               0,00         0,00         0,00          6          0
Signature · 3 lines
Tomek
http://wpkg.org
WPKG - software deployment and upgrades with Samba
#6 lore raw
In reply to Arjan van de Ven
Quoted 8 lines · Arjan van de Ven
Quoted 4 lines · Tomasz Chmielewski

1.00 1.10 1.06 1/65 782

This server is barely used, and as I remember, loadavg was always close to 0.00 on that system.

Quoted 6 lines

remember that load is the sum of running/runable processes and processes in D state (waiting for IO generally, but not always). I'm pretty sure your load comes from one of the later...

ps ought to tell you which one it is... (if not, an "echo t > /proc/sysrq-trigger" will dump the kernel state including the offending process, and will also tell us where exactly that process is)

Wohoo, you're great, that was it:

root     29547  0.0  0.3   7516   996 ?        D    Nov25   0:00 CROND
root     29548  0.0  0.3   7516   996 ?        Ss   Nov25   0:00 CROND

I stopped it, and loadavg is back to 0.

Now I have to figure out what CROND was doing...

Does ps always show processes in D state in CAPITAL letters?

After cron restart it is "crond", as usual.

Signature · 3 lines
Tomek
http://wpkg.org
WPKG - software deployment and upgrades with Samba
#7 lore raw
Quoted 3 lines

Now I have to figure out what CROND was doing...

if it does it again, do that echo t > /proc/sysrq-trigger and look for "crond" in the output, that will give a kernel backtrace of where crond was.

Quoted 2 lines

Does ps always show processes in D state in CAPITAL letters?

hmm I never noticed that before, but then again "ps" may be undergoing innovations ;)

#8 lore raw
In reply to Tomasz Chmielewski

On Thu, 2005-12-01 at 14:00 +0100, Tomasz Chmielewski wrote:
[snip]

Quoted 5 lines

Now I have to figure out what CROND was doing...

Does ps always show processes in D state in CAPITAL letters?

After cron restart it is "crond", as usual.

crond is the regular cron daemon. CROND is what cron names its child processes as they run scheduled commands.

I've seen cron stuck in D from running user crontabs on unavailable NFS mounts.

Signature · 1 line
Zan Lynx <[email protected]>