Skip to contentSkip to conversation

Runnable threads on run queue

DISCUSSION Ask List 6 participants 12 messages Jul 8, 2006 – Jul 13, 2006

lore
Share & tools
lore

Ask List

author
#0 lore raw

Have an issue maybe someone on this list can help with.

At times of very high load the number of processes on the run queue drops to
0 then jumps really high and then drops to 0 and back and forth. It seems to
last 10 seconds or so. If you look at this vmstat you can see an example of what I mean. Now im not a linux kernel expert but i am thinking it has something to do with the scheduling algorithm and locking of the run queue. For this particular application I need all available threads to be processed as fast as possible. Is there a way for me to elimnate this behavior or at least minimize the window in which there are no threads on the run queue? Is there a sysctl parameter I can use?

Please help.

procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----

 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
83  0   1328 301684  37868 1520632    0    0     0   264  400  1332 98  2  0  0
17  0   1328 293936  37868 1520688    0    0     0     0  537   979 97  3  0  0
73  0   1328 293688  37868 1520712    0    0     0     0  268  2643 98  2  0  0
80  0   1328 277220  37868 1520756    0    0     0     0  351   824 98  2  0  0
49  0   1328 262452  37868 1520800    0    0     0     0  393  1882 97  3  0  0
45  0   1328 246796  37868 1520828    0    0     0   304  302  1631 96  4  0  0
55  0   1328 243852  37868 1520872    0    0     0     0  356  1101 99  1  0  0
17  0   1328 228672  37868 1520916    0    0     0     0  336   748 97  3  0  0
 0  0   1328 299948  37868 1520956    0    0     0     0  299   821 78  3 19  0
 0  0   1328 299184  37868 1520960    0    0     0     0  168    78  8  0 92  0
 0  0   1328 299184  37868 1520960    0    0     0   248  173    38  0  1 99  0
 0  0   1328 299184  37868 1520960    0    0     0     0  160    20  0  0 100  0
 0  0   1328 299184  37868 1520960    0    0     0     0  151     6  0  0 100  0
 0  0   1328 299184  37868 1520960    0    0     0     0  162    42  0  1 99  0
 1  0   1328 299188  37868 1520960    0    0     0     0  161    24  0  0 100  0
 0  0   1328 298808  37868 1520988    0    0     0   100  303  1119 57  0 42  0
 0  0   1328 298808  37868 1520988    0    0     0     0  162    22  0  1 99  0
 3  0   1328 298808  37868 1520992    0    0     0     0  195   233 16  0 84  0
14  0   1328 298788  37868 1521032    0    0     0     0  400  1158 87  3 10  0
54  0   1328 298860  37868 1521064    0    0     0     0  438   940 97  3  0  0
80  0   1328 298296  37868 1521092    0    0     0   180  476   556 97  3  0  0
29  0   1328 294632  37868 1521148    0    0     0     0  824  1178 99  1  0  0
68  0   1328 292936  37868 1521172    0    0     0     0  404  2283 96  4  0  0
73  0   1328 292740  37868 1521216    0    0     0     0  521   828 98  2  0  0
38  0   1328 260340  37868 1521260    0    0     0     0  405  1069 96  4  0  0
46  0   1328 253072  37868 1521292    0    0     0   300  371  1692 95  5  0  0
71  0   1328 244084  37868 1521328    0    0     0     0  357  1478 98  2  0  0
71  0   1328 233916  37868 1521384    0    0     0     0  528  1121 97  3  0  0
32  0   1328 222784  37868 1521416    0    0     0     0  347  1191 96  4  0  0
76  0   1328 212396  37868 1521448    0    0     0     0  337  2526 97  3  0  0
71  0   1328 198684  37868 1521488    0    0     0   284  497   942 98  2  0  0
40  0   1328 189964  37868 1521532    0    0     0     0  420  1525 96  4  0  0
53  0   1328 179656  37868 1521576    0    0     0     0  391  1983 98  2  0  0
91  0   1328 169164  37868 1521608    0    0     0     0  415  2018 98  2  0  0
70  0   1328 151300  37868 1521648    0    0     0     0  411  1769 98  2  0  0
43  0   1328 145980  37868 1521684    0    0     0   308  420  1713 96  4  0  0
48  0   1328 142708  37868 1521724    0    0     0     0  290  1490 97  3  0  0
76  0   1328 126080  37868 1521752    0    0     0     0  389  1568 97  3  0  0
85  0   1328 120544  37864 1518164    0    0     0     0  365  1261 96  4  0  0
51  0   1328 121312  37864 1506908    0    0     0     0  306  1217 98  2  0  0
55  0   1328 121488  37864 1495128    0    0     0   292  364  1976 98  2  0  0
79  0   1328 120408  37864 1486072    0    0     0     0  328  2106 97  3  0  0
29  0   1328 216660  37864 1482744    0    0     0     0  387   866 97  3  0  0
 0  0   1328 321932  37864 1482788    0    0     0     0  289   750 67  3 31  0
 0  0   1328 321932  37864 1482788    0    0     0     0  158    10  0  0 100  0
 2  0   1328 321912  37864 1482792    0    0     0   268  201   156  4  1 94  0
 0  0   1328 321892  37864 1482796    0    0     0     0  180   270  7  0 93  0
 0  0   1328 321892  37864 1482796    0    0     0     0  152     4  0  0 100  0
 0  0   1328 321880  37864 1482796    0    0     0     0  158    26  0  1 99  0
 0  0   1328 321844  37864 1482820    0    0     0     0  330   454 41  1 58  0
 0  0   1328 321844  37864 1482820    0    0     0   120  167    30  0  0 100  0
 0  0   1328 321844  37864 1482820    0    0     0     0  166    35  1  0 99  0
35  0   1328 321476  37864 1482836    0    0     0     0  530  1026 67  2 31  0
76  0   1328 321528  37868 1482864    0    0     0     0  406  1744 96  4  0  0
41  0   1328 321172  37868 1482920    0    0     0   192  409   690 97  3  0  0
34  0   1328 314788  37868 1482956    0    0     0     0  356  1616 97  3  0  0
63  0   1328 314368  37868 1482996    0    0     0     0  437  1277 98  2  0  0
 1  0   1328 331744  37868 1483044    0    0     0     0  331   709 90  3  7  0
 0  0   1328 331724  37868 1483048    0    0     0     0  174   395  4  0 96  0
 0  0   1328 331724  37868 1483048    0    0     0   224  168    16  0  0 100  0
 0  0   1328 331724  37868 1483048    0    0     0     0  167    54  0  1 99  0
 7  0   1328 331744  37868 1483048    0    0     0     0  238   167 10  0 90  0
46  0   1328 330788  37868 1483076    0    0     0     0  878  1677 98  2  0  0
84  0   1328 330444  37868 1483100    0    0     0     0  425  1449 97  3  0  0
#1 lore raw
Quoted 11 lines · Ask List

Have an issue maybe someone on this list can help with.

At times of very high load the number of processes on the run queue drops to 0 then jumps really high and then drops to 0 and back and forth. It seems to last 10 seconds or so. If you look at this vmstat you can see an example of what I mean. Now im not a linux kernel expert but i am thinking it has something to do with the scheduling algorithm and locking of the run queue. For this particular application I need all available threads to be processed as fast as possible. Is there a way for me to elimnate this behavior or at least minimize the window in which there are no threads on the run queue? Is there a sysctl parameter I can use?

If there's a runnable task on the system, the run queue should never empty except inside schedule(). The scheduler should then swap expired and active.

First question - what kernel are you running? Is it stock?

Second question - what's the application? Are you sure your threads just aren't falling into interruptible sleep due to an app bug of some sort? Are you observing misbehavior in the application (long pauses) or just in the reporting?

Thanks,
Chase

#3 lore raw
In reply to Ask List
Quoted 1 line · * Ask List ([email protected]) wrote:

Have an issue maybe someone on this list can help with.

<snip>

Quoted 14 lines

Please help.

procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----

 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
83  0   1328 301684  37868 1520632    0    0     0   264  400  1332 98  2  0  0
17  0   1328 293936  37868 1520688    0    0     0     0  537   979 97  3  0  0
73  0   1328 293688  37868 1520712    0    0     0     0  268  2643 98  2  0  0
80  0   1328 277220  37868 1520756    0    0     0     0  351   824 98  2  0  0
49  0   1328 262452  37868 1520800    0    0     0     0  393  1882 97  3  0  0
45  0   1328 246796  37868 1520828    0    0     0   304  302  1631 96  4  0  0
55  0   1328 243852  37868 1520872    0    0     0     0  356  1101 99  1  0  0
17  0   1328 228672  37868 1520916    0    0     0     0  336   748 97  3  0  0
 0  0   1328 299948  37868 1520956    0    0     0     0  299   821 78  3 19  0
 0  0   1328 299184  37868 1520960    0    0     0     0  168    78  8  0 92  0

Could you also post the output of iostat -x 1 covering the same period? (You might need to restrict the set of devices if you have a lot) The pattern of bursts of output is something I've seen on apps just trying to do continuous large writes and I'm wondering what you are seeing there.

Dave

Signature · 4 lines
 -----Open up your eyes, open up your mind, open up your code -------   
/ Dr. David Alan Gilbert    | Running GNU/Linux on Alpha,68K| Happy  \ 
\ gro.gilbert @ treblig.org | MIPS,x86,ARM,SPARC,PPC & HPPA | In Hex /
 \ _________________________|_____ http://www.treblig.org   |_______/
author
#2 lore raw
In reply to Chase Venters
Quoted 19 lines · Chase Venters <chase.venters <at> clientec.com> writes:
Quoted 11 lines · Ask List

Have an issue maybe someone on this list can help with.

At times of very high load the number of processes on the run queue drops to 0 then jumps really high and then drops to 0 and back and forth. It seems to last 10 seconds or so. If you look at this vmstat you can see an example of what I mean. Now im not a linux kernel expert but i am thinking it has something to do with the scheduling algorithm and locking of the run queue. For this particular application I need all available threads to be processed as fast as possible. Is there a way for me to elimnate this behavior or at least minimize the window in which there are no threads on the run queue? Is there a sysctl parameter I can use?

If there's a runnable task on the system, the run queue should never empty except inside schedule(). The scheduler should then swap expired and active.

First question - what kernel are you running? Is it stock?

Quoted 8 lines

Second question - what's the application? Are you sure your threads just aren't falling into interruptible sleep due to an app bug of some sort? Are you observing misbehavior in the application (long pauses) or just in the reporting?

Thanks,
Chase

The kernel version is a debian kernel source version 2.4.27-3 and it was recompiled to support SMP, High Memory, etc. The application is SpamAssassin version 3.1.1. It is possible there may be an app bug, however I do not know this for certain. We have manipulated the configuration of the daemon to try and aleviate the symptoms to no avail. We experience the issues if we use a mysql backend for the bayes db or not. We are experiencing misbehavior in the application in the sense of the time it takes for messages to be processed. It normally takes tenths of a second to process incoming mail, however we notice the processing time jump to over 10 seconds each time the run queue drops to 0 and then drops back down to tenths of a second when the queue fills back up.

author
#4 lore raw
In reply to Dr. David Alan Gilbert

I dont exactly have iostat -x 1 from the same time frame. But I do have the collected sar data from before and during our period of high load. Here is a snippet before ...

tps rtps wtps bread/s bwrtn/s

11:46:00 PM      5.00      0.00      5.00      0.00    160.00
11:46:01 PM      8.91      0.00      8.91      0.00    182.18
11:46:02 PM     10.00      0.00     10.00      0.00    368.00
11:46:03 PM      7.92      0.00      7.92      0.00    142.57
11:46:04 PM     13.00      0.00     13.00      0.00    336.00
11:46:05 PM      9.90      0.00      9.90      0.00    261.39
11:46:06 PM      9.00      0.00      9.00      0.00    264.00
11:46:07 PM      6.93      0.00      6.93      0.00    198.02
11:46:08 PM      8.00      0.00      8.00      0.00    288.00
11:46:09 PM     12.87      0.00     12.87      0.00    324.75
11:46:10 PM      9.00      0.00      9.00      0.00    280.00
11:46:11 PM      6.93      0.00      6.93      0.00    134.65
11:46:12 PM     12.00      0.00     12.00      0.00    336.00
11:46:13 PM     10.89      0.00     10.89      0.00    253.47
11:46:14 PM     18.00      0.00     18.00      0.00    464.00
11:46:15 PM      4.81      0.00      4.81      0.00     84.62
11:46:16 PM     10.00      0.00     10.00      0.00    328.00
11:46:17 PM     10.89      0.00     10.89      0.00    269.31
11:46:18 PM     11.00      0.00     11.00      0.00    304.00
11:46:19 PM     30.69      0.00     30.69      0.00    451.49
11:46:20 PM      9.00      0.00      9.00      0.00    272.00
11:46:21 PM      5.94      0.00      5.94      0.00     95.05
11:46:22 PM     10.00      0.00     10.00      0.00    304.00
11:46:23 PM      5.94      0.00      5.94      0.00    150.50
11:46:24 PM     17.00      0.00     17.00      0.00    432.00
11:46:25 PM      6.93      0.00      6.93      0.00    190.10
11:46:26 PM     10.00      0.00     10.00      0.00    344.00
11:46:27 PM      8.91      0.00      8.91      0.00    166.34
11:46:28 PM      7.00      0.00      7.00      0.00    192.00
11:46:29 PM     15.84      0.00     15.84      0.00    427.72
11:46:30 PM      7.00      0.00      7.00      0.00    168.00
11:46:31 PM      9.90      0.00      9.90      0.00    221.78
11:46:32 PM     12.00      0.00     12.00      0.00    360.00
11:46:33 PM     10.89      0.00     10.89      0.00    245.54
11:46:34 PM     10.00      0.00     10.00      0.00    280.00
11:46:35 PM      6.93      0.00      6.93      0.00    134.65
11:46:36 PM     11.00      0.00     11.00      0.00    296.00
11:46:37 PM      8.91      0.00      8.91      0.00    205.94
11:46:38 PM     12.00      0.00     12.00      0.00    376.00
11:46:39 PM     14.85      0.00     14.85      0.00    435.64
11:46:40 PM      9.00      0.00      9.00      0.00    248.00
11:46:41 PM      7.92      0.00      7.92      0.00    237.62
11:46:42 PM     10.00      0.00     10.00      0.00    320.00
11:46:43 PM      5.94      0.00      5.94      0.00     55.45
11:46:44 PM     15.00      0.00     15.00      0.00    408.00
11:46:45 PM      9.90      0.00      9.90      0.00    229.70
11:46:46 PM     10.00      0.00     10.00      0.00    272.00
11:46:47 PM     10.89      0.00     10.89      0.00    269.31
11:46:48 PM     10.00      0.00     10.00      0.00    272.00
11:46:49 PM     36.63      0.00     36.63      0.00    514.85
11:46:50 PM     11.00      0.00     11.00      0.00    296.00
11:46:51 PM      8.91      0.00      8.91      0.00    205.94
11:46:52 PM     11.00      0.00     11.00      0.00    312.00
11:46:53 PM      8.91      0.00      8.91      0.00    190.10
11:46:54 PM     15.00      0.00     15.00      0.00    368.00
11:46:55 PM      9.90      0.00      9.90      0.00    253.47
11:46:56 PM     11.00      0.00     11.00      0.00    352.00
11:46:57 PM      8.91      0.00      8.91      0.00    245.54
11:46:58 PM      9.00      0.00      9.00      0.00    256.00
11:46:59 PM     11.88      0.00     11.88      0.00    308.91
11:47:00 PM      7.00      0.00      7.00      0.00    168.00

and here is a snippet during high load....

12:13:00 AM      6.00      0.00      6.00      0.00    224.00
12:13:01 AM      8.06      0.00      8.06      0.00    180.65
12:13:02 AM     18.00      0.00     18.00      0.00    544.00
12:13:03 AM      8.00      0.00      8.00      0.00    192.00
12:13:04 AM     47.00      0.00     47.00      0.00    856.00
12:13:05 AM      8.91      0.00      8.91      0.00    229.70
12:13:06 AM     15.00      0.00     15.00      0.00    392.00
12:13:07 AM      9.90      0.00      9.90      0.00    229.70
12:13:08 AM      8.00      0.00      8.00      0.00    232.00
12:13:09 AM     15.52      0.00     15.52      0.00    379.31
12:13:10 AM      6.98      0.00      6.98      0.00    198.45
12:13:12 AM     12.96      0.00     12.96      0.00    348.15
12:13:13 AM     17.00      0.00     17.00      0.00    424.00
12:13:14 AM     28.74      0.00     28.74      0.00    526.95
12:13:15 AM     13.46      0.00     13.46      0.00    361.54
12:13:16 AM      9.40      0.00      9.40      0.00    225.64
12:13:17 AM     15.00      0.00     15.00      0.00    488.00
12:13:19 AM     14.91      0.00     14.91      0.00    377.64
12:13:20 AM      9.00      0.00      9.00      0.00    296.00
12:13:21 AM     12.15      0.00     12.15      0.00    366.36
12:13:22 AM     26.00      0.00     26.00      0.00    784.00
12:13:24 AM     11.06      0.00     11.06      0.00    324.42
12:13:25 AM     14.81      0.00     14.81      0.00    333.33
12:13:26 AM     25.47      0.00     25.47      0.00    777.36
12:13:27 AM     19.00      0.00     19.00      0.00    480.00
12:13:28 AM     20.79      0.00     20.79      0.00    538.61
12:13:29 AM      5.00      0.00      5.00      0.00    136.00
12:13:31 AM     12.73      0.00     12.73      0.00    298.18
12:13:32 AM     23.00      0.00     23.00      0.00    632.00
12:13:33 AM     36.79      0.00     36.79      0.00   1011.32
12:13:34 AM     37.50      0.00     37.50      0.00    950.00
12:13:35 AM      7.76      0.00      7.76      0.00    186.21
12:13:36 AM     12.93      0.00     12.93      0.00    324.14
12:13:38 AM      8.57      0.00      8.57      0.00    210.29
12:13:39 AM     30.00      0.00     30.00      0.00    696.00
12:13:40 AM      9.90      0.00      9.90      0.00    245.54
12:13:41 AM     12.00      0.00     12.00      0.00    328.00
12:13:42 AM      5.94      0.00      5.94      0.00     63.37
12:13:43 AM      7.00      0.00      7.00      0.00    256.00
12:13:44 AM     44.54      0.00     44.54      0.00    746.22
12:13:45 AM      9.71      0.00      9.71      0.00    248.54
12:13:46 AM     13.89      0.00     13.89      0.00    370.37
12:13:47 AM     13.00      0.00     13.00      0.00    336.00
12:13:48 AM     13.86      0.00     13.86      0.00    324.75
12:13:49 AM     15.00      0.00     15.00      0.00    344.00
12:13:50 AM      3.96      0.00      3.96      0.00     39.60
12:13:51 AM     11.00      0.00     11.00      0.00    368.00
12:13:52 AM      7.92      0.00      7.92      0.00    174.26
12:13:54 AM     10.17      0.00     10.17      0.00    266.67
12:13:55 AM      7.41      0.00      7.41      0.00    133.33
12:13:56 AM     15.00      0.00     15.00      0.00    328.00
12:13:57 AM      5.71      0.00      5.71      0.00     91.43
12:13:58 AM      9.68      0.00      9.68      0.00    316.13
12:13:59 AM     24.27      0.00     24.27      0.00    520.39
12:14:00 AM     10.89      0.00     10.89      0.00    324.75

... I hope this helps.

#5 lore raw
In reply to Ask List
Quoted 11 lines · Ask List

procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----

 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
83  0   1328 301684  37868 1520632    0    0     0   264  400  1332 98  2  0  0
17  0   1328 293936  37868 1520688    0    0     0     0  537   979 97  3  0  0
73  0   1328 293688  37868 1520712    0    0     0     0  268  2643 98  2  0  0
80  0   1328 277220  37868 1520756    0    0     0     0  351   824 98  2  0  0
49  0   1328 262452  37868 1520800    0    0     0     0  393  1882 97  3  0  0
45  0   1328 246796  37868 1520828    0    0     0   304  302  1631 96  4  0  0
55  0   1328 243852  37868 1520872    0    0     0     0  356  1101 99  1  0  0
17  0   1328 228672  37868 1520916    0    0     0     0  336   748 97  3  0  0
 0  0   1328 299948  37868 1520956    0    0     0     0  299   821 78  3 19  0
Quoted 8 lines
 0  0   1328 299184  37868 1520960    0    0     0     0  168    78  8  0 92  0
 0  0   1328 299184  37868 1520960    0    0     0   248  173    38  0  1 99  0
 0  0   1328 299184  37868 1520960    0    0     0     0  160    20  0  0 100  0
 0  0   1328 299184  37868 1520960    0    0     0     0  151     6  0  0 100  0
 0  0   1328 299184  37868 1520960    0    0     0     0  162    42  0  1 99  0
 1  0   1328 299188  37868 1520960    0    0     0     0  161    24  0  0 100  0
 0  0   1328 298808  37868 1520988    0    0     0   100  303  1119 57  0 42  0
 0  0   1328 298808  37868 1520988    0    0     0     0  162    22  0  1 99  0

Looking at the interrupts column, I suspect you have a network problem, not a scheduler problem. Looks to me like your SpamAssasins are simply running out of work to do because your network traffic comes in bursts.

-Mike

#10 lore raw
In reply to Ask List
Quoted 5 lines · Ask List

Have an issue maybe someone on this list can help with.

At times of very high load the number of processes on the run queue drops to
0 then jumps really high and then drops to 0 and back and forth. It seems to
last 10 seconds or so.

Are you using sendmail by any chance? :)

We start out with a low load averag, so sendmail forks as many spamassassins as it can...

Quoted 12 lines

procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----

 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
83  0   1328 301684  37868 1520632    0    0     0   264  400  1332 98  2  0  0
17  0   1328 293936  37868 1520688    0    0     0     0  537   979 97  3  0  0
73  0   1328 293688  37868 1520712    0    0     0     0  268  2643 98  2  0  0
80  0   1328 277220  37868 1520756    0    0     0     0  351   824 98  2  0  0
49  0   1328 262452  37868 1520800    0    0     0     0  393  1882 97  3  0  0
45  0   1328 246796  37868 1520828    0    0     0   304  302  1631 96  4  0  0
55  0   1328 243852  37868 1520872    0    0     0     0  356  1101 99  1  0  0
17  0   1328 228672  37868 1520916    0    0     0     0  336   748 97  3  0  0
 0  0   1328 299948  37868 1520956    0    0     0     0  299   821 78  3 19  0
 0  0   1328 299184  37868 1520960    0    0     0     0  168    78  8  0 92  0

... and guess what?

The load average went through the roof, so sendmail stops forking spamassassins. Now nothing is running, and sendmail will not start forking new spamassassins again until after the load average has decayed to an acceptable level.

After that, it will fork way too many at once again, and the load average will go through the roof. Lather, rinse, repeat.

You'd probably be better off limiting the number of simultaneous local mail deliveries to something reasonable, so the load average always stays at an acceptable level - and more importantly, all of the CPU capacity could be used if needed...

Signature · 3 lines
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan
#6 lore raw
In reply to Mike Galbraith
Quoted 3 lines · Mike Galbraith
Quoted 2 lines · Ask List

procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa

[...]

Quoted 3 lines

Looking at the interrupts column, I suspect you have a network problem, not a scheduler problem. Looks to me like your SpamAssasins are simply running out of work to do because your network traffic comes in bursts.

spamassassin acted up here some time ago. With personal training and some messages it went to a loop and the load went through the roof. Couldn't find a cure, plus some hundred users with large personalized rule files were causing problems anyway, so we axed that.

Signature · 4 lines
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513
author
#11 lore raw
In reply to Rik van Riel

We are not running sendmail. We developed our own mail server in-house. We have a cluster of these mail servers sending spam traffic to a cluster of SA servers and we use the round-robin parameter when starting the spamd process and start the daemon with a ton of min/spare/max children. So we dont see the forking issue you mention.

author
#7 lore raw
In reply to Mike Galbraith

Mike Galbraith <efault <at> gmx.de> writes:
...

Quoted 7 lines

Looking at the interrupts column, I suspect you have a network problem, not a scheduler problem. Looks to me like your SpamAssasins are simply running out of work to do because your network traffic comes in bursts.

-Mike

Network Problem? So your saying our mail servers are not sending spam traffic fast enough if spam assassin processes are running out of work to do? So when our mail servers are not sending spam traffic we see our cpu,cs,interrupts, & runnable threads drop ...?

I'd really like to believe this is true, however in the sa logs there are still plenty of B (busy threads)...

#8 lore raw
Quoted 4 lines · Ask List

Network Problem? So your saying our mail servers are not sending spam traffic fast enough if spam assassin processes are running out of work to do? So when our mail servers are not sending spam traffic we see our cpu,cs,interrupts, & runnable threads drop ...?

More or less, yes. I think somebody is dropping the communication ball.

-Mike

author
#9 lore raw

I'll look into it. Thanks for the input.