Skip to contentSkip to conversation

System state too high for too long...

DISCUSSION Dan A. Dickey 2 participants 5 messages Jun 7, 2005 – Jun 9, 2005

lore
Share & tools
lore
#0 lore raw

This problem has now been persistent enough in the last few kernels I've run that I've subscribed (once again) to the linux-kernel list and would like to report it.
I'm using gentoo-sources-2.6.11-r9.

When the system is compiling something, the state typically stays at about 85-95% system time. This just really does not seem right for my workload, and additionally only appeared a few releases ago (sorry, I didn't bother to track it - I thought it might go away in a release or two; but it has not).

Here is a little output of 'vmstat 5' when this is happening:
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us
sy id wa

 1  0  12752  61548  57252 211092    0    0     2    50 1083   810 11 
89  0  0
 1  0  12752  57572  57320 211160    0    0     0    63 1089   683  9 
91  0  0
 1  0  12752  63288  57328 211220    0    0     8    39 1084   765 11 
89  0  0
 1  0  12752  60648  57348 211200    0    0     0    56 1086   647  6 
94  0  0
 1  0  12752  54972  57348 211200    0    0     0     3 1079   659  8 
92  0  0
 1  0  12752  62284  57348 211268    0    0     4    53 1087   807 17 
83  0  0
 1  0  12752  59400  57356 211328    0    0     0    34 1222  1919 17 
83  0  0

Can someone help me to debug this further? Thanks.
-Dan

Signature · 5 lines
Dan A. Dickey
[email protected]

SAVVIS
Transforming Information Technology
#1 lore raw
Quoted 17 lines · Dan A. Dickey

This problem has now been persistent enough in the last few kernels
I've run that I've subscribed (once again) to the linux-kernel list and would like to report it.
I'm using gentoo-sources-2.6.11-r9.

When the system is compiling something, the state typically stays at about 85-95% system time. This just really does not seem right for my workload, and additionally only appeared a few releases ago (sorry, I didn't bother to track it - I thought it might go away in a release or two; but it has not).

Here is a little output of 'vmstat 5' when this is happening:
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us
sy id wa

Quoted 14 lines
  1  0  12752  61548  57252 211092    0    0     2    50 1083   810 11 
 89  0  0
  1  0  12752  57572  57320 211160    0    0     0    63 1089   683  9 
 91  0  0
  1  0  12752  63288  57328 211220    0    0     8    39 1084   765 11 
 89  0  0
  1  0  12752  60648  57348 211200    0    0     0    56 1086   647  6 
 94  0  0
  1  0  12752  54972  57348 211200    0    0     0     3 1079   659  8 
 92  0  0
  1  0  12752  62284  57348 211268    0    0     4    53 1087   807 17 
 83  0  0
  1  0  12752  59400  57356 211328    0    0     0    34 1222  1919 17 
 83  0  0

(grr, wordwrapping.)

- Check that your disks are running in DMA mode (if they're IDE) (with hdparm)

- See what `time make' says

- Generate a kernel profile (See Documentation/basic_profiling.txt)

#2 lore raw
Quoted 5 lines · Andrew Morton
Quoted 4 lines · Dan A. Dickey

This problem has now been persistent enough in the last few kernels I've run that I've subscribed (once again) to the linux-kernel list and would like to report it.
I'm using gentoo-sources-2.6.11-r9.

Switched to vanilla-sources 2.6.12-rc6 just to rule out the possible influence of patches...

Quoted 5 lines
Quoted 5 lines

When the system is compiling something, the state typically stays at about 85-95% system time. This just really does not seem right for my workload, and additionally only appeared a few releases ago (sorry, I didn't bother to track it - I
thought it might go away in a release or two; but it has not).

...

Quoted 6 lines
Quoted 4 lines

1 0 12752 62284 57348 211268 0 0 4 53 1087
807 17 83 0 0
1 0 12752 59400 57356 211328 0 0 0 34 1222
1919 17 83 0 0

(grr, wordwrapping.)

Yeah; sorry - my email gui is doing that. I turned it off. Don't know why I had it on in the first place.

Quoted 2 lines

- Check that your disks are running in DMA mode (if they're IDE) (with hdparm)

It looks to me like it is:

# hdparm -I /dev/hda

/dev/hda:

ATA device, with non-removable media

        Model Number:       WDC WD400BB-75DEA0
        Serial Number:      WD-WMAD16857838
        Firmware Revision:  05.03E05

Standards:
Supported: 5 4 3 2
Likely used: 6
Configuration:

        Logical         max     current
        cylinders       16383   16383
        heads           16      16
        sectors/track   63      63
        --
        CHS current addressable sectors:   16514064
        LBA    user addressable sectors:   78165360
        device size with M = 1024*1024:       38166 MBytes
        device size with M = 1000*1000:       40020 MBytes (40 GB)

Capabilities:
LBA, IORDY(can be disabled)
bytes avail on r/w long: 40 Queue depth: 1
Standby timer values: spec'd by Standard, with device specific minimum R/W multiple sector transfer: Max = 16 Current = 16 Recommended acoustic management value: 128, current value: 254
DMA: mdma0 mdma1 mdma2 udma0 udma1 udma2 udma3 udma4 *udma5
Cycle time: min=120ns recommended=120ns
PIO: pio0 pio1 pio2 pio3 pio4
Cycle time: no flow control=120ns IORDY flow control=120ns
Commands/features:
Enabled Supported:
* READ BUFFER cmd
* WRITE BUFFER cmd
* Host Protected Area feature set
* Look-ahead
* Write cache
* Power Management feature set
* SMART feature set
* Device Configuration Overlay feature set
* Automatic Acoustic Management feature set
SET MAX security extension
* DOWNLOAD MICROCODE cmd
* SMART self-test
* SMART error logging
HW reset results:
CBLID- above Vih
Device num = 0 determined by CSEL
Checksum: correct

Quoted 1 line

- See what `time make' says

It says:

real    9m51.034s
user    1m43.371s
sys     6m2.018s
Quoted 1 line

- Generate a kernel profile (See Documentation/basic_profiling.txt)

How much of it is useful? I used readprofile; here is what to me looks like the most relevant portion of it.

# sort -nr +2 < captured_profile | head -n 40
281609 kmem_cache_free                          2448.7739
 25158 default_idle                             535.2766
  8704 kernel_map_pages                          93.5914
  3870 sock_poll                                 86.0000
  6133 poison_obj                                80.6974
  8846 kfree                                     60.5890
  2784 _atomic_dec_and_lock                      30.2609
   696 fput                                      27.8400
  2824 change_page_attr                          24.9912
  2483 fget                                      22.9907
  9986 buffered_rmqueue                          18.6306
  2086 sysenter_past_esp                         17.8291
  1091 sub_preempt_count                         17.3175
  5607 __d_lookup                                14.9920
  1295 strncpy_from_user                         11.6667
   568 add_preempt_count                         11.3600
  1597 vfs_getattr                               10.9384
  3672 tcp_poll                                  10.2857
  1270 __copy_to_user_ll                         10.0794
  1323 __do_softirq                               9.4500
   902 kmap_atomic                                7.9823
   290 dbg_redzone1                               6.9048
   731 __get_zone_counts                          6.7064
   417 path_release                               6.6190
   277 kmem_flagcheck                             6.2955
   136 __mod_page_state                           6.1818
    84 obj_reallen                                6.0000
   947 kmem_cache_alloc                           5.9937
   995 getname                                    5.5587
   182 kunmap_atomic                              5.2000
   139 get_offset_tsc                             5.1481
    65 ide_outbsync                               5.0000
  1824 path_lookup                                4.9973
   410 find_get_page                              4.6591
   429 find_vma                                   4.5638
   749 __might_sleep                              4.4850
  4216 do_wp_page                                 4.4756
   245 sys_lstat64                                4.3750
   553 follow_mount                               3.6867
   623 page_address                               3.4611

If more of this would help, let me know and I can get it to whomever can help. Thanks!
-Dan

Signature · 5 lines
Dan A. Dickey
[email protected]

SAVVIS
Transforming Information Technology
#3 lore raw
Quoted 2 lines · Dan A. Dickey

8704 kernel_map_pages 93.5914

Ah. CONFIG_DEBUG_PAGEALLOC can be most expensive. Please disable it and try again.

#4 lore raw
Quoted 6 lines · Andrew Morton
Quoted 2 lines · Dan A. Dickey

8704 kernel_map_pages 93.5914

Ah. CONFIG_DEBUG_PAGEALLOC can be most expensive. Please disable it and try again.

Andrew,
Thank you very much! That appears to have been it. I think it may have crept into my config way back when when I enabled kernel debugging so I could get the Alt-SysRq stuff enabled. I didn't realize I had enabled other things to be turned on as well. This certainly taught me to take a closer look at my config when changing it. Thanks again.
My system is much more speedy at compiles now!
-Dan

Signature · 5 lines
Dan A. Dickey
[email protected]

SAVVIS
Transforming Information Technology