Runnable threads on run queue
Have an issue maybe someone on this list can help with.
At times of very high load the number of processes on the run queue drops to
0 then jumps really high and then drops to 0 and back and forth. It seems to
last 10 seconds or so. If you look at this vmstat you can see an example of what I mean. Now im not a linux kernel expert but i am thinking it has something to do with the scheduling algorithm and locking of the run queue. For this particular application I need all available threads to be processed as fast as possible. Is there a way for me to elimnate this behavior or at least minimize the window in which there are no threads on the run queue? Is there a sysctl parameter I can use?
Please help.
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa 83 0 1328 301684 37868 1520632 0 0 0 264 400 1332 98 2 0 0 17 0 1328 293936 37868 1520688 0 0 0 0 537 979 97 3 0 0 73 0 1328 293688 37868 1520712 0 0 0 0 268 2643 98 2 0 0 80 0 1328 277220 37868 1520756 0 0 0 0 351 824 98 2 0 0 49 0 1328 262452 37868 1520800 0 0 0 0 393 1882 97 3 0 0 45 0 1328 246796 37868 1520828 0 0 0 304 302 1631 96 4 0 0 55 0 1328 243852 37868 1520872 0 0 0 0 356 1101 99 1 0 0 17 0 1328 228672 37868 1520916 0 0 0 0 336 748 97 3 0 0 0 0 1328 299948 37868 1520956 0 0 0 0 299 821 78 3 19 0 0 0 1328 299184 37868 1520960 0 0 0 0 168 78 8 0 92 0 0 0 1328 299184 37868 1520960 0 0 0 248 173 38 0 1 99 0 0 0 1328 299184 37868 1520960 0 0 0 0 160 20 0 0 100 0 0 0 1328 299184 37868 1520960 0 0 0 0 151 6 0 0 100 0 0 0 1328 299184 37868 1520960 0 0 0 0 162 42 0 1 99 0 1 0 1328 299188 37868 1520960 0 0 0 0 161 24 0 0 100 0 0 0 1328 298808 37868 1520988 0 0 0 100 303 1119 57 0 42 0 0 0 1328 298808 37868 1520988 0 0 0 0 162 22 0 1 99 0 3 0 1328 298808 37868 1520992 0 0 0 0 195 233 16 0 84 0 14 0 1328 298788 37868 1521032 0 0 0 0 400 1158 87 3 10 0 54 0 1328 298860 37868 1521064 0 0 0 0 438 940 97 3 0 0 80 0 1328 298296 37868 1521092 0 0 0 180 476 556 97 3 0 0 29 0 1328 294632 37868 1521148 0 0 0 0 824 1178 99 1 0 0 68 0 1328 292936 37868 1521172 0 0 0 0 404 2283 96 4 0 0 73 0 1328 292740 37868 1521216 0 0 0 0 521 828 98 2 0 0 38 0 1328 260340 37868 1521260 0 0 0 0 405 1069 96 4 0 0 46 0 1328 253072 37868 1521292 0 0 0 300 371 1692 95 5 0 0 71 0 1328 244084 37868 1521328 0 0 0 0 357 1478 98 2 0 0 71 0 1328 233916 37868 1521384 0 0 0 0 528 1121 97 3 0 0 32 0 1328 222784 37868 1521416 0 0 0 0 347 1191 96 4 0 0 76 0 1328 212396 37868 1521448 0 0 0 0 337 2526 97 3 0 0 71 0 1328 198684 37868 1521488 0 0 0 284 497 942 98 2 0 0 40 0 1328 189964 37868 1521532 0 0 0 0 420 1525 96 4 0 0 53 0 1328 179656 37868 1521576 0 0 0 0 391 1983 98 2 0 0 91 0 1328 169164 37868 1521608 0 0 0 0 415 2018 98 2 0 0 70 0 1328 151300 37868 1521648 0 0 0 0 411 1769 98 2 0 0 43 0 1328 145980 37868 1521684 0 0 0 308 420 1713 96 4 0 0 48 0 1328 142708 37868 1521724 0 0 0 0 290 1490 97 3 0 0 76 0 1328 126080 37868 1521752 0 0 0 0 389 1568 97 3 0 0 85 0 1328 120544 37864 1518164 0 0 0 0 365 1261 96 4 0 0 51 0 1328 121312 37864 1506908 0 0 0 0 306 1217 98 2 0 0 55 0 1328 121488 37864 1495128 0 0 0 292 364 1976 98 2 0 0 79 0 1328 120408 37864 1486072 0 0 0 0 328 2106 97 3 0 0 29 0 1328 216660 37864 1482744 0 0 0 0 387 866 97 3 0 0 0 0 1328 321932 37864 1482788 0 0 0 0 289 750 67 3 31 0 0 0 1328 321932 37864 1482788 0 0 0 0 158 10 0 0 100 0 2 0 1328 321912 37864 1482792 0 0 0 268 201 156 4 1 94 0 0 0 1328 321892 37864 1482796 0 0 0 0 180 270 7 0 93 0 0 0 1328 321892 37864 1482796 0 0 0 0 152 4 0 0 100 0 0 0 1328 321880 37864 1482796 0 0 0 0 158 26 0 1 99 0 0 0 1328 321844 37864 1482820 0 0 0 0 330 454 41 1 58 0 0 0 1328 321844 37864 1482820 0 0 0 120 167 30 0 0 100 0 0 0 1328 321844 37864 1482820 0 0 0 0 166 35 1 0 99 0 35 0 1328 321476 37864 1482836 0 0 0 0 530 1026 67 2 31 0 76 0 1328 321528 37868 1482864 0 0 0 0 406 1744 96 4 0 0 41 0 1328 321172 37868 1482920 0 0 0 192 409 690 97 3 0 0 34 0 1328 314788 37868 1482956 0 0 0 0 356 1616 97 3 0 0 63 0 1328 314368 37868 1482996 0 0 0 0 437 1277 98 2 0 0 1 0 1328 331744 37868 1483044 0 0 0 0 331 709 90 3 7 0 0 0 1328 331724 37868 1483048 0 0 0 0 174 395 4 0 96 0 0 0 1328 331724 37868 1483048 0 0 0 224 168 16 0 0 100 0 0 0 1328 331724 37868 1483048 0 0 0 0 167 54 0 1 99 0 7 0 1328 331744 37868 1483048 0 0 0 0 238 167 10 0 90 0 46 0 1328 330788 37868 1483076 0 0 0 0 878 1677 98 2 0 0 84 0 1328 330444 37868 1483100 0 0 0 0 425 1449 97 3 0 0
Quoted 11 lines · Ask List
Have an issue maybe someone on this list can help with.
At times of very high load the number of processes on the run queue drops to 0 then jumps really high and then drops to 0 and back and forth. It seems to last 10 seconds or so. If you look at this vmstat you can see an example of what I mean. Now im not a linux kernel expert but i am thinking it has something to do with the scheduling algorithm and locking of the run queue. For this particular application I need all available threads to be processed as fast as possible. Is there a way for me to elimnate this behavior or at least minimize the window in which there are no threads on the run queue? Is there a sysctl parameter I can use?
If there's a runnable task on the system, the run queue should never empty except inside schedule(). The scheduler should then swap expired and active.
First question - what kernel are you running? Is it stock?
Second question - what's the application? Are you sure your threads just aren't falling into interruptible sleep due to an app bug of some sort? Are you observing misbehavior in the application (long pauses) or just in the reporting?
Thanks,
Chase
Quoted 19 lines · Chase Venters <chase.venters <at> clientec.com> writes:
Quoted 11 lines · Ask List
Have an issue maybe someone on this list can help with.
At times of very high load the number of processes on the run queue drops to 0 then jumps really high and then drops to 0 and back and forth. It seems to last 10 seconds or so. If you look at this vmstat you can see an example of what I mean. Now im not a linux kernel expert but i am thinking it has something to do with the scheduling algorithm and locking of the run queue. For this particular application I need all available threads to be processed as fast as possible. Is there a way for me to elimnate this behavior or at least minimize the window in which there are no threads on the run queue? Is there a sysctl parameter I can use?
If there's a runnable task on the system, the run queue should never empty except inside
schedule(). The scheduler should then swap expired and active.First question - what kernel are you running? Is it stock?
Quoted 8 lines
Second question - what's the application? Are you sure your threads just aren't falling into interruptible sleep due to an app bug of some sort? Are you observing misbehavior in the application (long pauses) or just in the reporting?
Thanks,
Chase
The kernel version is a debian kernel source version 2.4.27-3 and it was recompiled to support SMP, High Memory, etc. The application is SpamAssassin version 3.1.1. It is possible there may be an app bug, however I do not know this for certain. We have manipulated the configuration of the daemon to try and aleviate the symptoms to no avail. We experience the issues if we use a mysql backend for the bayes db or not. We are experiencing misbehavior in the application in the sense of the time it takes for messages to be processed. It normally takes tenths of a second to process incoming mail, however we notice the processing time jump to over 10 seconds each time the run queue drops to 0 and then drops back down to tenths of a second when the queue fills back up.
Quoted 1 line · * Ask List ([email protected]) wrote:
Have an issue maybe someone on this list can help with.
<snip>
Quoted 14 lines
Please help.
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa 83 0 1328 301684 37868 1520632 0 0 0 264 400 1332 98 2 0 0 17 0 1328 293936 37868 1520688 0 0 0 0 537 979 97 3 0 0 73 0 1328 293688 37868 1520712 0 0 0 0 268 2643 98 2 0 0 80 0 1328 277220 37868 1520756 0 0 0 0 351 824 98 2 0 0 49 0 1328 262452 37868 1520800 0 0 0 0 393 1882 97 3 0 0 45 0 1328 246796 37868 1520828 0 0 0 304 302 1631 96 4 0 0 55 0 1328 243852 37868 1520872 0 0 0 0 356 1101 99 1 0 0 17 0 1328 228672 37868 1520916 0 0 0 0 336 748 97 3 0 0 0 0 1328 299948 37868 1520956 0 0 0 0 299 821 78 3 19 0 0 0 1328 299184 37868 1520960 0 0 0 0 168 78 8 0 92 0
Could you also post the output of iostat -x 1 covering the same period? (You might need to restrict the set of devices if you have a lot) The pattern of bursts of output is something I've seen on apps just trying to do continuous large writes and I'm wondering what you are seeing there.
Dave
Signature · 4 lines
-----Open up your eyes, open up your mind, open up your code ------- / Dr. David Alan Gilbert | Running GNU/Linux on Alpha,68K| Happy \ \ gro.gilbert @ treblig.org | MIPS,x86,ARM,SPARC,PPC & HPPA | In Hex / \ _________________________|_____ http://www.treblig.org |_______/
I dont exactly have iostat -x 1 from the same time frame. But I do have the collected sar data from before and during our period of high load. Here is a snippet before ...
tps rtps wtps bread/s bwrtn/s
11:46:00 PM 5.00 0.00 5.00 0.00 160.00 11:46:01 PM 8.91 0.00 8.91 0.00 182.18 11:46:02 PM 10.00 0.00 10.00 0.00 368.00 11:46:03 PM 7.92 0.00 7.92 0.00 142.57 11:46:04 PM 13.00 0.00 13.00 0.00 336.00 11:46:05 PM 9.90 0.00 9.90 0.00 261.39 11:46:06 PM 9.00 0.00 9.00 0.00 264.00 11:46:07 PM 6.93 0.00 6.93 0.00 198.02 11:46:08 PM 8.00 0.00 8.00 0.00 288.00 11:46:09 PM 12.87 0.00 12.87 0.00 324.75 11:46:10 PM 9.00 0.00 9.00 0.00 280.00 11:46:11 PM 6.93 0.00 6.93 0.00 134.65 11:46:12 PM 12.00 0.00 12.00 0.00 336.00 11:46:13 PM 10.89 0.00 10.89 0.00 253.47 11:46:14 PM 18.00 0.00 18.00 0.00 464.00 11:46:15 PM 4.81 0.00 4.81 0.00 84.62 11:46:16 PM 10.00 0.00 10.00 0.00 328.00 11:46:17 PM 10.89 0.00 10.89 0.00 269.31 11:46:18 PM 11.00 0.00 11.00 0.00 304.00 11:46:19 PM 30.69 0.00 30.69 0.00 451.49 11:46:20 PM 9.00 0.00 9.00 0.00 272.00 11:46:21 PM 5.94 0.00 5.94 0.00 95.05 11:46:22 PM 10.00 0.00 10.00 0.00 304.00 11:46:23 PM 5.94 0.00 5.94 0.00 150.50 11:46:24 PM 17.00 0.00 17.00 0.00 432.00 11:46:25 PM 6.93 0.00 6.93 0.00 190.10 11:46:26 PM 10.00 0.00 10.00 0.00 344.00 11:46:27 PM 8.91 0.00 8.91 0.00 166.34 11:46:28 PM 7.00 0.00 7.00 0.00 192.00 11:46:29 PM 15.84 0.00 15.84 0.00 427.72 11:46:30 PM 7.00 0.00 7.00 0.00 168.00 11:46:31 PM 9.90 0.00 9.90 0.00 221.78 11:46:32 PM 12.00 0.00 12.00 0.00 360.00 11:46:33 PM 10.89 0.00 10.89 0.00 245.54 11:46:34 PM 10.00 0.00 10.00 0.00 280.00 11:46:35 PM 6.93 0.00 6.93 0.00 134.65 11:46:36 PM 11.00 0.00 11.00 0.00 296.00 11:46:37 PM 8.91 0.00 8.91 0.00 205.94 11:46:38 PM 12.00 0.00 12.00 0.00 376.00 11:46:39 PM 14.85 0.00 14.85 0.00 435.64 11:46:40 PM 9.00 0.00 9.00 0.00 248.00 11:46:41 PM 7.92 0.00 7.92 0.00 237.62 11:46:42 PM 10.00 0.00 10.00 0.00 320.00 11:46:43 PM 5.94 0.00 5.94 0.00 55.45 11:46:44 PM 15.00 0.00 15.00 0.00 408.00 11:46:45 PM 9.90 0.00 9.90 0.00 229.70 11:46:46 PM 10.00 0.00 10.00 0.00 272.00 11:46:47 PM 10.89 0.00 10.89 0.00 269.31 11:46:48 PM 10.00 0.00 10.00 0.00 272.00 11:46:49 PM 36.63 0.00 36.63 0.00 514.85 11:46:50 PM 11.00 0.00 11.00 0.00 296.00 11:46:51 PM 8.91 0.00 8.91 0.00 205.94 11:46:52 PM 11.00 0.00 11.00 0.00 312.00 11:46:53 PM 8.91 0.00 8.91 0.00 190.10 11:46:54 PM 15.00 0.00 15.00 0.00 368.00 11:46:55 PM 9.90 0.00 9.90 0.00 253.47 11:46:56 PM 11.00 0.00 11.00 0.00 352.00 11:46:57 PM 8.91 0.00 8.91 0.00 245.54 11:46:58 PM 9.00 0.00 9.00 0.00 256.00 11:46:59 PM 11.88 0.00 11.88 0.00 308.91 11:47:00 PM 7.00 0.00 7.00 0.00 168.00
and here is a snippet during high load....
12:13:00 AM 6.00 0.00 6.00 0.00 224.00 12:13:01 AM 8.06 0.00 8.06 0.00 180.65 12:13:02 AM 18.00 0.00 18.00 0.00 544.00 12:13:03 AM 8.00 0.00 8.00 0.00 192.00 12:13:04 AM 47.00 0.00 47.00 0.00 856.00 12:13:05 AM 8.91 0.00 8.91 0.00 229.70 12:13:06 AM 15.00 0.00 15.00 0.00 392.00 12:13:07 AM 9.90 0.00 9.90 0.00 229.70 12:13:08 AM 8.00 0.00 8.00 0.00 232.00 12:13:09 AM 15.52 0.00 15.52 0.00 379.31 12:13:10 AM 6.98 0.00 6.98 0.00 198.45 12:13:12 AM 12.96 0.00 12.96 0.00 348.15 12:13:13 AM 17.00 0.00 17.00 0.00 424.00 12:13:14 AM 28.74 0.00 28.74 0.00 526.95 12:13:15 AM 13.46 0.00 13.46 0.00 361.54 12:13:16 AM 9.40 0.00 9.40 0.00 225.64 12:13:17 AM 15.00 0.00 15.00 0.00 488.00 12:13:19 AM 14.91 0.00 14.91 0.00 377.64 12:13:20 AM 9.00 0.00 9.00 0.00 296.00 12:13:21 AM 12.15 0.00 12.15 0.00 366.36 12:13:22 AM 26.00 0.00 26.00 0.00 784.00 12:13:24 AM 11.06 0.00 11.06 0.00 324.42 12:13:25 AM 14.81 0.00 14.81 0.00 333.33 12:13:26 AM 25.47 0.00 25.47 0.00 777.36 12:13:27 AM 19.00 0.00 19.00 0.00 480.00 12:13:28 AM 20.79 0.00 20.79 0.00 538.61 12:13:29 AM 5.00 0.00 5.00 0.00 136.00 12:13:31 AM 12.73 0.00 12.73 0.00 298.18 12:13:32 AM 23.00 0.00 23.00 0.00 632.00 12:13:33 AM 36.79 0.00 36.79 0.00 1011.32 12:13:34 AM 37.50 0.00 37.50 0.00 950.00 12:13:35 AM 7.76 0.00 7.76 0.00 186.21 12:13:36 AM 12.93 0.00 12.93 0.00 324.14 12:13:38 AM 8.57 0.00 8.57 0.00 210.29 12:13:39 AM 30.00 0.00 30.00 0.00 696.00 12:13:40 AM 9.90 0.00 9.90 0.00 245.54 12:13:41 AM 12.00 0.00 12.00 0.00 328.00 12:13:42 AM 5.94 0.00 5.94 0.00 63.37 12:13:43 AM 7.00 0.00 7.00 0.00 256.00 12:13:44 AM 44.54 0.00 44.54 0.00 746.22 12:13:45 AM 9.71 0.00 9.71 0.00 248.54 12:13:46 AM 13.89 0.00 13.89 0.00 370.37 12:13:47 AM 13.00 0.00 13.00 0.00 336.00 12:13:48 AM 13.86 0.00 13.86 0.00 324.75 12:13:49 AM 15.00 0.00 15.00 0.00 344.00 12:13:50 AM 3.96 0.00 3.96 0.00 39.60 12:13:51 AM 11.00 0.00 11.00 0.00 368.00 12:13:52 AM 7.92 0.00 7.92 0.00 174.26 12:13:54 AM 10.17 0.00 10.17 0.00 266.67 12:13:55 AM 7.41 0.00 7.41 0.00 133.33 12:13:56 AM 15.00 0.00 15.00 0.00 328.00 12:13:57 AM 5.71 0.00 5.71 0.00 91.43 12:13:58 AM 9.68 0.00 9.68 0.00 316.13 12:13:59 AM 24.27 0.00 24.27 0.00 520.39 12:14:00 AM 10.89 0.00 10.89 0.00 324.75
... I hope this helps.
Quoted 11 lines · Ask List
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa 83 0 1328 301684 37868 1520632 0 0 0 264 400 1332 98 2 0 0 17 0 1328 293936 37868 1520688 0 0 0 0 537 979 97 3 0 0 73 0 1328 293688 37868 1520712 0 0 0 0 268 2643 98 2 0 0 80 0 1328 277220 37868 1520756 0 0 0 0 351 824 98 2 0 0 49 0 1328 262452 37868 1520800 0 0 0 0 393 1882 97 3 0 0 45 0 1328 246796 37868 1520828 0 0 0 304 302 1631 96 4 0 0 55 0 1328 243852 37868 1520872 0 0 0 0 356 1101 99 1 0 0 17 0 1328 228672 37868 1520916 0 0 0 0 336 748 97 3 0 0 0 0 1328 299948 37868 1520956 0 0 0 0 299 821 78 3 19 0
Quoted 8 lines
0 0 1328 299184 37868 1520960 0 0 0 0 168 78 8 0 92 0 0 0 1328 299184 37868 1520960 0 0 0 248 173 38 0 1 99 0 0 0 1328 299184 37868 1520960 0 0 0 0 160 20 0 0 100 0 0 0 1328 299184 37868 1520960 0 0 0 0 151 6 0 0 100 0 0 0 1328 299184 37868 1520960 0 0 0 0 162 42 0 1 99 0 1 0 1328 299188 37868 1520960 0 0 0 0 161 24 0 0 100 0 0 0 1328 298808 37868 1520988 0 0 0 100 303 1119 57 0 42 0 0 0 1328 298808 37868 1520988 0 0 0 0 162 22 0 1 99 0
Looking at the interrupts column, I suspect you have a network problem, not a scheduler problem. Looks to me like your SpamAssasins are simply running out of work to do because your network traffic comes in bursts.
-Mike
Quoted 3 lines · Mike Galbraith
Quoted 2 lines · Ask List
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
[...]
Quoted 3 lines
Looking at the interrupts column, I suspect you have a network problem, not a scheduler problem. Looks to me like your SpamAssasins are simply running out of work to do because your network traffic comes in bursts.
spamassassin acted up here some time ago. With personal training and some messages it went to a loop and the load went through the roof. Couldn't find a cure, plus some hundred users with large personalized rule files were causing problems anyway, so we axed that.
Signature · 4 lines
Dr. Horst H. von Brand User #22616 counter.li.org Departamento de Informatica Fono: +56 32 654431 Universidad Tecnica Federico Santa Maria +56 32 654239 Casilla 110-V, Valparaiso, Chile Fax: +56 32 797513
Mike Galbraith <efault <at> gmx.de> writes:
...
Quoted 7 lines
Looking at the interrupts column, I suspect you have a network problem, not a scheduler problem. Looks to me like your SpamAssasins are simply running out of work to do because your network traffic comes in bursts.
-Mike
Network Problem? So your saying our mail servers are not sending spam traffic fast enough if spam assassin processes are running out of work to do? So when our mail servers are not sending spam traffic we see our cpu,cs,interrupts, & runnable threads drop ...?
I'd really like to believe this is true, however in the sa logs there are still plenty of B (busy threads)...
Quoted 4 lines · Ask List
Network Problem? So your saying our mail servers are not sending spam traffic fast enough if spam assassin processes are running out of work to do? So when our mail servers are not sending spam traffic we see our cpu,cs,interrupts, & runnable threads drop ...?
More or less, yes. I think somebody is dropping the communication ball.
-Mike
I'll look into it. Thanks for the input.
Quoted 5 lines · Ask List
Have an issue maybe someone on this list can help with.
At times of very high load the number of processes on the run queue drops to
0 then jumps really high and then drops to 0 and back and forth. It seems to
last 10 seconds or so.
Are you using sendmail by any chance? :)
We start out with a low load averag, so sendmail forks as many spamassassins as it can...
Quoted 12 lines
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa 83 0 1328 301684 37868 1520632 0 0 0 264 400 1332 98 2 0 0 17 0 1328 293936 37868 1520688 0 0 0 0 537 979 97 3 0 0 73 0 1328 293688 37868 1520712 0 0 0 0 268 2643 98 2 0 0 80 0 1328 277220 37868 1520756 0 0 0 0 351 824 98 2 0 0 49 0 1328 262452 37868 1520800 0 0 0 0 393 1882 97 3 0 0 45 0 1328 246796 37868 1520828 0 0 0 304 302 1631 96 4 0 0 55 0 1328 243852 37868 1520872 0 0 0 0 356 1101 99 1 0 0 17 0 1328 228672 37868 1520916 0 0 0 0 336 748 97 3 0 0 0 0 1328 299948 37868 1520956 0 0 0 0 299 821 78 3 19 0 0 0 1328 299184 37868 1520960 0 0 0 0 168 78 8 0 92 0
... and guess what?
The load average went through the roof, so sendmail stops forking spamassassins. Now nothing is running, and sendmail will not start forking new spamassassins again until after the load average has decayed to an acceptable level.
After that, it will fork way too many at once again, and the load average will go through the roof. Lather, rinse, repeat.
You'd probably be better off limiting the number of simultaneous local mail deliveries to something reasonable, so the load average always stays at an acceptable level - and more importantly, all of the CPU capacity could be used if needed...
Signature · 3 lines
"Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan
We are not running sendmail. We developed our own mail server in-house. We have a cluster of these mail servers sending spam traffic to a cluster of SA servers and we use the round-robin parameter when starting the spamd process and start the daemon with a ton of min/spare/max children. So we dont see the forking issue you mention.