pipe and af/unix latency differences between aa and jam on smp
The -jam patchset is interesting because it starts out with the entire -aa patchset and adds a few things.
Sometimes small differences in LMbench between -jam and -aa are just CPU bounces on SMP. The difference for pipe and af/unix latency only appears on SMP too, but it is very consistent. (My k6/2 has small differences between -aa and -jam for pipe and af/unix latency).
You will know better what could make the difference:
This is the averages:
*Local* Communication latencies in microseconds - smaller is better
-------------------------------------------------------------------
kernel Pipe AF/Unix
----------------- ------- -------
2.4.19-pre10-aa4 33.941 70.216
2.4.19-pre10-jam2 7.877 16.699
These are the individual runs:
*Local* Communication latencies in microseconds - smaller is better
-------------------------------------------------------------------
OS Pipe AF/Unix
----------------------------- ------- ------ Linux 2.4.19-pre10-aa4 33.999 73.024 Linux 2.4.19-pre10-aa4 35.829 73.261 Linux 2.4.19-pre10-aa4 16.710 74.830 Linux 2.4.19-pre10-aa4 37.221 66.354 Linux 2.4.19-pre10-aa4 36.259 68.433 Linux 2.4.19-pre10-aa4 36.429 68.215 Linux 2.4.19-pre10-aa4 35.379 77.147 Linux 2.4.19-pre10-aa4 29.300 73.641 Linux 2.4.19-pre10-aa4 35.798 64.875 Linux 2.4.19-pre10-aa4 35.691 75.433 Linux 2.4.19-pre10-aa4 35.372 73.398 Linux 2.4.19-pre10-aa4 33.516 69.183 Linux 2.4.19-pre10-aa4 34.986 69.254 Linux 2.4.19-pre10-aa4 33.743 69.893 Linux 2.4.19-pre10-aa4 32.679 71.900 Linux 2.4.19-pre10-aa4 34.131 71.812 Linux 2.4.19-pre10-aa4 33.444 72.454 Linux 2.4.19-pre10-aa4 36.531 71.956 Linux 2.4.19-pre10-aa4 37.838 69.731 Linux 2.4.19-pre10-aa4 34.359 71.522 Linux 2.4.19-pre10-aa4 33.286 71.609 Linux 2.4.19-pre10-aa4 32.361 43.533 Linux 2.4.19-pre10-aa4 31.716 74.131 Linux 2.4.19-pre10-aa4 35.218 72.001 Linux 2.4.19-pre10-aa4 36.709 67.795
Linux 2.4.19-pre10-jam2 7.9977 14.495 Linux 2.4.19-pre10-jam2 7.8406 14.044 Linux 2.4.19-pre10-jam2 7.7899 14.006 Linux 2.4.19-pre10-jam2 7.8584 13.819 Linux 2.4.19-pre10-jam2 7.8379 14.453 Linux 2.4.19-pre10-jam2 7.8781 14.156 Linux 2.4.19-pre10-jam2 7.8881 14.238 Linux 2.4.19-pre10-jam2 7.9833 14.168 Linux 2.4.19-pre10-jam2 7.7772 78.765 Linux 2.4.19-pre10-jam2 8.0816 13.703 Linux 2.4.19-pre10-jam2 7.8605 14.042 Linux 2.4.19-pre10-jam2 7.7982 13.883 Linux 2.4.19-pre10-jam2 7.6362 14.286 Linux 2.4.19-pre10-jam2 7.7480 13.989 Linux 2.4.19-pre10-jam2 7.9262 13.947 Linux 2.4.19-pre10-jam2 8.0904 14.014 Linux 2.4.19-pre10-jam2 7.8480 14.310 Linux 2.4.19-pre10-jam2 7.7982 14.171 Linux 2.4.19-pre10-jam2 7.9776 14.234 Linux 2.4.19-pre10-jam2 7.7931 14.125 Linux 2.4.19-pre10-jam2 7.8553 14.110 Linux 2.4.19-pre10-jam2 7.7294 14.285 Linux 2.4.19-pre10-jam2 8.3361 14.131 Linux 2.4.19-pre10-jam2 7.7797 14.039 Linux 2.4.19-pre10-jam2 7.8265 14.043
For pipe and af/unix bandwidth, the difference appears to just be a CPU bounce here and there.
jam patchsets are at:
http://giga.cps.unizar.es/~magallon/linux/
--
Randy Hron
http://home.earthlink.net/~rwhron/kernel/bigbox.html
Quoted 11 lines · On 2002.07.09 [email protected] wrote:
The -jam patchset is interesting because it starts out with the entire -aa patchset and adds a few things.
Sometimes small differences in LMbench between -jam and -aa are just CPU bounces on SMP. The difference for pipe and af/unix latency only appears on SMP too, but it is very consistent. (My k6/2 has small differences between -aa and -jam for pipe and af/unix latency).
You will know better what could make the difference:
Quoted 9 lines
This is the averages:
*Local* Communication latencies in microseconds - smaller is better
-------------------------------------------------------------------
kernel Pipe AF/Unix
----------------- ------- -------
2.4.19-pre10-aa4 33.941 70.216
2.4.19-pre10-jam2 7.877 16.699
Candidates in pre10-jam2 could be:
11-irqbalance-B1.bz2
12-smptimers-A0.bz2
13-irqrate-A1.bz2
excluding anything that has nothing to do with pipes or latency.
Could you try latest -rc1-aa2 ? It includes also irqbalance, so it could be on varable less in the equation.
I dropped smptimers and irqrate because they did not mix very well with bproc and O1 scheduler, but I can try to add them again.
I have a rc1-jam2 ready, but the only important change wrt SMP could be the mem-barrier specific implementation for P3/P4, and your box is an AMD.
??
Signature · 4 lines
J.A. Magallon \ Software is like sex: It's better when it's free mailto:[email protected] \ -- Linus Torvalds, FSF T-shirt Linux werewolf 2.4.19-rc1-jam2, Mandrake Linux 8.3 (Cooker) for i586 gcc (GCC) 3.1.1 (Mandrake Linux 8.3 3.1.1-0.7mdk)
Quoted 5 lines · On 2002.07.09 J.A. Magallon wrote:
I have a rc1-jam2 ready, but the only important change wrt SMP could be the mem-barrier specific implementation for P3/P4, and your box is an AMD.
Opps, I remembered your tests are done on a Quad Xeon ?
Signature · 4 lines
J.A. Magallon \ Software is like sex: It's better when it's free mailto:[email protected] \ -- Linus Torvalds, FSF T-shirt Linux werewolf 2.4.19-rc1-jam2, Mandrake Linux 8.3 (Cooker) for i586 gcc (GCC) 3.1.1 (Mandrake Linux 8.3 3.1.1-0.7mdk)
Quoted 1 line · J.A. Magallon
Opps, I remembered your tests are done on a Quad Xeon ?
Out of interest, is that a P4/Xeon?
Cheers,
Zwane Mwaikambo
Signature · 1 line
function.linuxpower.ca
Quoted 11 lines · On 2002.07.09 [email protected] wrote:
The -jam patchset is interesting because it starts out with the entire -aa patchset and adds a few things.
Sometimes small differences in LMbench between -jam and -aa are just CPU bounces on SMP. The difference for pipe and af/unix latency only appears on SMP too, but it is very consistent. (My k6/2 has small differences between -aa and -jam for pipe and af/unix latency).
You will know better what could make the difference:
Quoted 9 lines
This is the averages:
*Local* Communication latencies in microseconds - smaller is better
-------------------------------------------------------------------
kernel Pipe AF/Unix
----------------- ------- -------
2.4.19-pre10-aa4 33.941 70.216
2.4.19-pre10-jam2 7.877 16.699
I took a look at your numbers:
*Local* Communication latencies in microseconds - smaller is better
-------------------------------------------------------------------
kernel Pipe AF/Unix UDP RPC/UDP TCP RPC/TCP TCPconn ----------------------------- ------- ------- ------- ------- ------- ------- ------- 2.4.19-pre7-jam6 29.513 42.369 58.6165 60.7792 50.2572 82.4976 87.321 2.4.19-pre8-jam2 7.697 15.274 59.6730 60.8190 55.276 82.1297 89.416 2.4.19-pre8-jam2-nowuos 7.739 14.929 57.9326 60.5497 55.9745 81.8908 90.370
(last line says that wake-up-sync is not responsible...)
Main changes between first two were irqbalance and ide6->ide10.
Signature · 4 lines
J.A. Magallon \ Software is like sex: It's better when it's free mailto:[email protected] \ -- Linus Torvalds, FSF T-shirt Linux werewolf 2.4.19-rc1-jam2, Mandrake Linux 8.3 (Cooker) for i586 gcc (GCC) 3.1.1 (Mandrake Linux 8.3 3.1.1-0.7mdk)
Quoted 8 lines · On Mon, 8 Jul 2002 [email protected] wrote:
Sometimes small differences in LMbench between -jam and -aa are just CPU bounces on SMP. The difference for pipe and af/unix latency only appears on SMP too, but it is very consistent. (My k6/2 has small differences between -aa and -jam for pipe and af/unix latency).
You will know better what could make the difference:
Quoted 8 lines
This is the averages:
*Local* Communication latencies in microseconds - smaller is better
-------------------------------------------------------------------
kernel Pipe AF/Unix
----------------- ------- -------
2.4.19-pre10-aa4 33.941 70.216
2.4.19-pre10-jam2 7.877 16.699
Small differences? The only thing I would call small is the latency of the jam kernel!
If (a) this is a real value which results in ~5x latency reduction in non-benchmark applications, and (b) doesn't have some resulting penalty (there are some free lunches in Linux), then it would be desirable.
I have an IPC test which measures time for a datum to move form process A to process B and back, using various methods, I'll try to build these kernels and test it in my next free day. I'd love to test latency of SysV message queues as well, since these turn out to be good solutions to some types of N:M client-server problems.
Signature · 3 lines
bill davidsen <[email protected]> CTO, TMR Associates, Inc Doing interesting things with little computers since 1979.