NFS , Kjournald - status D (uninterruptible sleep) - maquina lenta.
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
Dears, I've got a double dual xeon wich sometimes is getting too slow.
While i was observing one of these slowly times, I realized the time wait of processors come close to 100% and nfsd and kjournald process become to D status (uniterruptible sleep) .Does someone know why this can be happening or how can I troubleshoot it?
Below is a 'draw' of my top at one of these moments.
[root@cromo src]# top
top - 10:59:02 up 6 days, 19:30, 17 users, load average: 14.77, 13.46, 13.00
Tasks: 189 total, 1 running, 188 sleeping, 0 stopped, 0 zombie Cpu0 : 0.0% us, 0.4% sy, 0.0% ni, 0.2% id, 98.5% wa, 0.0% hi, 0.9% si Cpu1 : 0.0% us, 0.4% sy, 0.0% ni, 0.0% id, 99.6% wa, 0.0% hi, 0.0% si Cpu2 : 0.0% us, 0.2% sy, 0.0% ni, 7.8% id, 92.0% wa, 0.0% hi, 0.0% si Cpu3 : 0.2% us, 0.8% sy, 0.0% ni, 0.0% id, 98.7% wa, 0.2% hi, 0.0% si Mem: 4025540k total, 4018264k used, 7276k free, 7908k buffers Swap: 4096564k total, 3288k used, 4093276k free, 3553252k cached
3941 root 15 0 0 0 0 D 2 0.0 80:29.30 nfsd
3938 root 15 0 0 0 0 D 1 0.0 81:02.02 nfsd
2459 root 15 0 0 0 0 D 1 0.0 30:57.04 kjournald
3937 root 15 0 0 0 0 D 1 0.0 82:07.99 nfsd
96 root 15 0 0 0 0 S 0 0.0 34:12.48 kswapd0
3940 root 15 0 0 0 0 D 0 0.0 80:22.55 nfsd
3942 root 15 0 0 0 0 D 0 0.0 80:23.66 nfsd
3939 root 15 0 0 0 0 D 0 0.0 82:22.55 nfsd
4283 root 16 0 20936 8136 1276 S 0 0.2 0:44.47 hald
21289 denismpa 16 0 6280 1092 776 R 0 0.0 0:00.12 top
1 root 16 0 4752 552 460 S 0 0.0 0:01.28 init
2 root RT 0 0 0 0 S 0 0.0 0:00.21 migration/0
3 root 34 19 0 0 0 S 0 0.0 0:00.05 ksoftirqd/0
4 root RT 0 0 0 0 S 0 0.0 0:00.28 migration/1
5 root 34 19 0 0 0 S 0 0.0 0:00.09 ksoftirqd/1
6 root RT 0 0 0 0 S 0 0.0 0:00.15 migration/2
7 root 34 19 0 0 0 S 0 0.0 0:00.05 ksoftirqd/2
8 root RT 0 0 0 0 S 0 0.0 0:00.15 migration/3
9 root 34 19 0 0 0 S 0 0.0 0:00.06 ksoftirqd/3
10 root 5 -10 0 0 0 S 0 0.0 0:01.01 events/0
11 root 5 -10 0 0 0 S 0 0.0 0:00.11 events/1
12 root 5 -10 0 0 0 S 0 0.0 0:00.06 events/2
13 root 5 -10 0 0 0 S 0 0.0 0:00.12 events/3
14 root 5 -10 0 0 0 S 0 0.0 0:00.00 khelper
15 root 15 -10 0 0 0 S 0 0.0 0:00.00 kacpid
63 root 5 -10 0 0 0 S 0 0.0 0: 00.00 kblockd/0
64 root 5 -10 0 0 0 S 0 0.0 0:00.00 kblockd/1
65 root 5 -10 0 0 0 S 0 0.0 0:00.00 kblockd/2
66 root 5 -10 0 0 0 S 0 0.0 0: 00.00 kblockd/3
67 root 15 0 0 0 0 S 0 0.0 0:00.06 khubd
97 root 12 -10 0 0 0 S 0 0.0 0:00.00 aio/0
98 root 5 -10 0 0 0 S 0 0.0 0:00.00 aio/1
99 root 5 -10 0 0 0 S 0 0.0 0:00.00 aio/2
100 root 5 -10 0 0 0 S 0 0.0 0:00.00 aio/3
244 root 23 0 0 0 0 S 0 0.0 0:00.00 kseriod
359 root 20 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_0
360 root 15 0 0 0 0 S 0 0.0 0:00.00 aacraid
379 root 5 -10 0 0 0 S 0 0.0 0:00.00 ata/0
380 root 5 -10 0 0 0 S 0 0.0 0: 00.00 ata/1
381 root 5 -10 0 0 0 S 0 0.0 0:00.00 ata/2
382 root 8 -10 0 0 0 S 0 0.0 0:00.00 ata/3
388 root 20 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_1
389 root 21 0 0 0 0 S 0 0.0 0:00.00 scsi_eh_2
403 root 15 0 0 0 0 D 0 0.0 0:03.87 kjournald
1708 root 6 -10 3604 444 364 S 0 0.0 0:00.00 udevdThanks and regards
--
Denis Anjos.
Cisco Certified Network Associate.
Universidade Federal do ABC
Santo André - SP - BR
Quoted 10 lines
Dears, I've got a double dual xeon wich sometimes is getting too slow.
While i was observing one of these slowly times, I realized the time wait of processors come close to 100% and nfsd and kjournald process become to D status (uniterruptible sleep) .Does someone know why this can be happening or how can I troubleshoot it?
Below is a 'draw' of my top at one of these moments.
Quoted 7 lines
[root@cromo src]# top
top - 10:59:02 up 6 days, 19:30, 17 users, load average: 14.77, 13.46, 13.00Tasks: 189 total, 1 running, 188 sleeping, 0 stopped, 0 zombie Cpu0 : 0.0% us, 0.4% sy, 0.0% ni, 0.2% id, 98.5% wa, 0.0% hi, 0.9% si Cpu1 : 0.0% us, 0.4% sy, 0.0% ni, 0.0% id, 99.6% wa, 0.0% hi, 0.0% si Cpu2 : 0.0% us, 0.2% sy, 0.0% ni, 7.8% id, 92.0% wa, 0.0% hi, 0.0% si Cpu3 : 0.2% us, 0.8% sy, 0.0% ni, 0.0% id, 98.7% wa, 0.2% hi, 0.0% si
All processors are in iowait - i.e. waiting for IO.
Quoted 11 lines
Mem: 4025540k total, 4018264k used, 7276k free, 7908k buffers Swap: 4096564k total, 3288k used, 4093276k free, 3553252k cached3941 root 15 0 0 0 0 D 2 0.0 80:29.30 nfsd 3938 root 15 0 0 0 0 D 1 0.0 81:02.02 nfsd 2459 root 15 0 0 0 0 D 1 0.0 30:57.04 kjournald 3937 root 15 0 0 0 0 D 1 0.0 82:07.99 nfsd 96 root 15 0 0 0 0 S 0 0.0 34:12.48 kswapd0 3940 root 15 0 0 0 0 D 0 0.0 80:22.55 nfsd 3942 root 15 0 0 0 0 D 0 0.0 80:23.66 nfsd 3939 root 15 0 0 0 0 D 0 0.0 82:22.55 nfsd
It seems like heavy disk/network activity. You can try running
vmstat 1
It will print every second the situation of your memory / IO
subsystem. You should especially see the ---io--- column ('bi' means blocks in, 'bo' blocks outputted). I guess they'd contain some non-trivial values...
BTW: I guess the processes wake up eventually, don't they?
Honza
Signature · 2 lines
Jan Kara <[email protected]> SuSE CR Labs
2007/12/17, Jan Kara <[email protected]>:
Quoted 38 lines
Quoted 10 lines
Dears, I've got a double dual xeon wich sometimes is getting too slow.
While i was observing one of these slowly times, I realized the time wait of processors come close to 100% and nfsd and kjournald process become to D status (uniterruptible sleep) .Does someone know why this can be happening or how can I troubleshoot it?
Below is a 'draw' of my top at one of these moments.
Quoted 7 lines
[root@cromo src]# top
top - 10:59:02 up 6 days, 19:30, 17 users, load average: 14.77, 13.46, 13.00Tasks: 189 total, 1 running, 188 sleeping, 0 stopped, 0 zombie Cpu0 : 0.0% us, 0.4% sy, 0.0% ni, 0.2% id, 98.5% wa, 0.0% hi, 0.9% si Cpu1 : 0.0% us, 0.4% sy, 0.0% ni, 0.0% id, 99.6% wa, 0.0% hi, 0.0% si Cpu2 : 0.0% us, 0.2% sy, 0.0% ni, 7.8% id, 92.0% wa, 0.0% hi, 0.0% si Cpu3 : 0.2% us, 0.8% sy, 0.0% ni, 0.0% id, 98.7% wa, 0.2% hi, 0.0% siAll processors are in iowait - i.e. waiting for IO.
Quoted 11 lines
Mem: 4025540k total, 4018264k used, 7276k free, 7908k buffers Swap: 4096564k total, 3288k used, 4093276k free, 3553252k cached3941 root 15 0 0 0 0 D 2 0.0 80:29.30 nfsd 3938 root 15 0 0 0 0 D 1 0.0 81:02.02 nfsd 2459 root 15 0 0 0 0 D 1 0.0 30:57.04 kjournald 3937 root 15 0 0 0 0 D 1 0.0 82:07.99 nfsd 96 root 15 0 0 0 0 S 0 0.0 34:12.48 kswapd0 3940 root 15 0 0 0 0 D 0 0.0 80:22.55 nfsd 3942 root 15 0 0 0 0 D 0 0.0 80:23.66 nfsd 3939 root 15 0 0 0 0 D 0 0.0 82:22.55 nfsdIt seems like heavy disk/network activity. You can try running
vmstat 1
It will print every second the situation of your memory / IO
subsystem. You should especially see the ---io--- column ('bi' means blocks in, 'bo' blocks outputted). I guess they'd contain some non-trivial values...
Quoted 5 lines
Honza
--
Jan Kara <[email protected]>
SuSE CR Labs
Hello, Jan.
The output of my vmstat look like this:
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa 0 1 3612 239600 2828 3364300 0 0 1494 1300 10 4 0 3 80 17 0 1 3612 187756 2872 3413420 0 0 44 24608 4274 10679 0 12 72 17 0 1 3612 153196 2900 3446100 0 0 28 17248 3349 6018 0 7 72 20 0 4 3612 134804 2908 3453980 0 0 8 61140 1795 4528 0 3 31 65 1 0 3612 144592 2920 3458796 0 0 4 18972 1689 2102 0 2 37 62 0 2 3612 86936 2980 3512048 0 0 56 26656 4554 14563 0 13 69 18 0 2 3612 45424 3024 3552532 0 0 40 19076 3774 8148 0 9 71 20 0 1 3612 15792 3048 3579368 0 0 44 25696 4430 11954 0 12 72 16 0 4 3612 7892 3064 3576972 0 0 16 74808 2430 12205 0 6 57 37 0 1 3612 17828 3080 3581512 0 0 8 18832 2423 6117 0 5 50 45 1 0 3612 16944 3068 3581048 0 0 40 26816 4126 15803 0 12 66 22 0 0 3612 16032 3128 3578200 0 0 60 32640 5137 23987 0 15 68 17 1 0 3612 15664 3144 3576620 0 0 56 33200 5178 17395 0 15 71 14 0 5 3612 7956 3160 3573272 0 0 20 74092 2077 5504 0 4 54 41 0 6 3612 12212 3168 3578568 0 0 8 30936 1964 3848 0 3 42 55 1 1 3612 15752 3200 3579760 0 0 24 22352 2975 9692 0 8 60 32 0 1 3612 16696 3240 3577340 0 0 40 22496 3954 15224 0 12 71 17 2 0 3612 15472 3212 3576824 0 0 40 26624 4466 14382 0 13 71 16 1 3 3612 5908 3228 3575924 0 0 16 72304 2328 8965 0 5 39 55 0 1 3612 14188 3244 3579512 0 0 8 17892 2169 4147 0 4 57 39 0 2 3612 17136 3292 3575520 0 0 56 28100 4672 16158 0 15 69 16
So, the bo, appears to be the problem.
Quoted 1 line
BTW: I guess the processes wake up eventually, don't they?
Yes, they eventually wake up for a while and then get in initerruptible sleep again
Considering that the BO is the problem, what can I try to fix it?
Thanks,
Signature · 4 lines
Denis Anjos. Cisco Certified Network Associate. Universidade Federal do ABC Santo André - SP - BR
Quoted 65 lines · Denis
2007/12/17, Jan Kara <[email protected]>:
Quoted 30 lines
Quoted 10 lines
Dears, I've got a double dual xeon wich sometimes is getting too slow.
While i was observing one of these slowly times, I realized the time wait of processors come close to 100% and nfsd and kjournald process become to D status (uniterruptible sleep) .Does someone know why this can be happening or how can I troubleshoot it?
Below is a 'draw' of my top at one of these moments.
Quoted 7 lines
[root@cromo src]# top
top - 10:59:02 up 6 days, 19:30, 17 users, load average: 14.77, 13.46, 13.00Tasks: 189 total, 1 running, 188 sleeping, 0 stopped, 0 zombie Cpu0 : 0.0% us, 0.4% sy, 0.0% ni, 0.2% id, 98.5% wa, 0.0% hi, 0.9% si Cpu1 : 0.0% us, 0.4% sy, 0.0% ni, 0.0% id, 99.6% wa, 0.0% hi, 0.0% si Cpu2 : 0.0% us, 0.2% sy, 0.0% ni, 7.8% id, 92.0% wa, 0.0% hi, 0.0% si Cpu3 : 0.2% us, 0.8% sy, 0.0% ni, 0.0% id, 98.7% wa, 0.2% hi, 0.0% siAll processors are in iowait - i.e. waiting for IO.
Quoted 11 lines
Mem: 4025540k total, 4018264k used, 7276k free, 7908k buffers Swap: 4096564k total, 3288k used, 4093276k free, 3553252k cached3941 root 15 0 0 0 0 D 2 0.0 80:29.30 nfsd 3938 root 15 0 0 0 0 D 1 0.0 81:02.02 nfsd 2459 root 15 0 0 0 0 D 1 0.0 30:57.04 kjournald 3937 root 15 0 0 0 0 D 1 0.0 82:07.99 nfsd 96 root 15 0 0 0 0 S 0 0.0 34:12.48 kswapd0 3940 root 15 0 0 0 0 D 0 0.0 80:22.55 nfsd 3942 root 15 0 0 0 0 D 0 0.0 80:23.66 nfsd 3939 root 15 0 0 0 0 D 0 0.0 82:22.55 nfsdQuoted 6 lines
It seems like heavy disk/network activity. You can try running
vmstat 1
It will print every second the situation of your memory / IO
subsystem. You should especially see the ---io--- column ('bi' means blocks in, 'bo' blocks outputted). I guess they'd contain some non-trivial values...The output of my vmstat look like this:
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa 0 1 3612 239600 2828 3364300 0 0 1494 1300 10 4 0 3 80 17 0 1 3612 187756 2872 3413420 0 0 44 24608 4274 10679 0 12 72 17 0 1 3612 153196 2900 3446100 0 0 28 17248 3349 6018 0 7 72 20 0 4 3612 134804 2908 3453980 0 0 8 61140 1795 4528 0 3 31 65 1 0 3612 144592 2920 3458796 0 0 4 18972 1689 2102 0 2 37 62 0 2 3612 86936 2980 3512048 0 0 56 26656 4554 14563 0 13 69 18 0 2 3612 45424 3024 3552532 0 0 40 19076 3774 8148 0 9 71 20 0 1 3612 15792 3048 3579368 0 0 44 25696 4430 11954 0 12 72 16 0 4 3612 7892 3064 3576972 0 0 16 74808 2430 12205 0 6 57 37 0 1 3612 17828 3080 3581512 0 0 8 18832 2423 6117 0 5 50 45 1 0 3612 16944 3068 3581048 0 0 40 26816 4126 15803 0 12 66 22 0 0 3612 16032 3128 3578200 0 0 60 32640 5137 23987 0 15 68 17 1 0 3612 15664 3144 3576620 0 0 56 33200 5178 17395 0 15 71 14 0 5 3612 7956 3160 3573272 0 0 20 74092 2077 5504 0 4 54 41 0 6 3612 12212 3168 3578568 0 0 8 30936 1964 3848 0 3 42 55 1 1 3612 15752 3200 3579760 0 0 24 22352 2975 9692 0 8 60 32 0 1 3612 16696 3240 3577340 0 0 40 22496 3954 15224 0 12 71 17 2 0 3612 15472 3212 3576824 0 0 40 26624 4466 14382 0 13 71 16 1 3 3612 5908 3228 3575924 0 0 16 72304 2328 8965 0 5 39 55 0 1 3612 14188 3244 3579512 0 0 8 17892 2169 4147 0 4 57 39 0 2 3612 17136 3292 3575520 0 0 56 28100 4672 16158 0 15 69 16
Quoted 1 line
So, the bo, appears to be the problem.
Yes, that means that your system is constantly writing about 30 MB/s to
disk. That's not too much but in case those writes are not sequential it can quite easily saturate your disk (depends on what disks you have)...
Quoted 5 lines
Quoted 1 line
BTW: I guess the processes wake up eventually, don't they?
Yes, they eventually wake up for a while and then get in initerruptible sleep again
Considering that the BO is the problem, what can I try to fix it?
First, you could find out what is writing those data to disk (I guess
the load is comming over NFS since nfsds are waiting for IO too). If you'd expect higher throughput from your disk subsystem, you can also verify that the disk is seeking a lot (for example by blktrace).
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR