lmbench results for 2.4 and 2.5
My previous testing with unix sockets prompted me to do a few lmbench runs with 2.4.19 and 2.5.65. The results have me a bit concerned, as there is no area where 2.5 is faster and several where it is significantly slower.
In particular:
stat is 8 times worse
open/close are 7 times worse
fork is twice as expensive
tcp latency is 5 times worse
file deletion and mmap are both twice as expensive tcp bandwidth is 5 times worse
Optimizing for muliple processors and heavy loads is nice, but this looks like its happening at the cost of basic performance. Is this really the route we should be taking?
L M B E N C H 2 . 0 S U M M A R Y
------------------------------------
Processor, Processes - times in microseconds - smaller is better
----------------------------------------------------------------
Host OS Mhz null null open selct sig sig fork exec sh
call I/O stat clos TCP inst hndl proc proc proc
------ ------------- ---- ---- ---- ---- ---- ----- ---- ---- ---- ---- ----
doug Linux 2.5.65 750 0.38 0.61 39.8 42.1 1.07 5.29 424. 2378 20.K doug Linux 2.5.65 750 0.38 0.54 40.2 44.2 1.07 5.31 439. 2386 20.K doug Linux 2.4.19 750 0.37 0.52 5.21 6.78 36.7 0.93 3.59 197. 1472 15.K
Context switching - times in microseconds - smaller is better
-------------------------------------------------------------
Host OS 2p/0K 2p/16K 2p/64K 8p/16K 8p/64K 16p/16K 16p/64K
ctxsw ctxsw ctxsw ctxsw ctxsw ctxsw ctxsw
--------- ------------- ----- ------ ------ ------ ------ ------- -------
doug Linux 2.5.65 1.790 3.0300 118.7 46.1 158.3 46.5 158.2 doug Linux 2.5.65 1.950 2.9800 122.6 46.3 159.5 47.1 158.7 doug Linux 2.4.19 1.690 2.6700 92.9 44.4 155.2 45.0 155.8
*Local* Communication latencies in microseconds - smaller is better
-------------------------------------------------------------------
Host OS 2p/0K Pipe AF UDP RPC/ TCP RPC/ TCP
ctxsw UNIX UDP TCP conn
--------- ------------- ----- ----- ---- ----- ----- ----- ----- ----
doug Linux 2.5.65 1.790 8.926 16.3 29.7 60.6 171.5 204.6 216. doug Linux 2.5.65 1.950 9.695 18.1 28.6 59.8 173.4 207.0 212. doug Linux 2.4.19 1.690 6.146 12.4 17.8 44.2 26.2 66.6 101.
File & VM system latencies in microseconds - smaller is better
--------------------------------------------------------------
Host OS 0K File 10K File Mmap Prot Page
Create Delete Create Delete Latency Fault Fault
--------- ------------- ------ ------ ------ ------ ------- ----- -----
doug Linux 2.5.65 110.2 65.0 242.5 100.7 3130.0 0.621 4.00000
doug Linux 2.5.65 110.1 63.5 237.2 96.6 3284.0 0.741 4.00000
doug Linux 2.4.19 82.5 32.4 187.5 47.9 1660.0 1.177 3.00000*Local* Communication bandwidths in MB/s - bigger is better
-----------------------------------------------------------
Host OS Pipe AF TCP File Mmap Bcopy Bcopy Mem Mem
UNIX reread reread (libc) (hand) read write
--------- ------------- ---- ---- ---- ------ ------ ------ ------ ---- -----
doug Linux 2.5.65 167. 94.7 14.3 212.5 354.8 214.5 215.9 474. 328.4 doug Linux 2.5.65 175. 86.3 14.2 216.3 354.1 211.4 210.9 474. 328.8 doug Linux 2.4.19 220. 108. 86.4 238.2 369.1 215.5 215.0 496. 328.0
Signature · 4 lines
Chris Friesen | MailStop: 043/33/F10 Nortel Networks | work: (613) 765-0557 3500 Carling Avenue | fax: (613) 765-2986 Nepean, ON K2H 8E9 Canada | email: [email protected]
Quoted 3 lines · Chris Friesen
My previous testing with unix sockets prompted me to do a few lmbench runs with 2.4.19 and 2.5.65. The results have me a bit concerned, as there is no area where 2.5 is faster and several where it is significantly slower.
Are you building both with SMP off, and pre-empt off ? Also both with APM/ACPI off ?
Quoted 5 lines · Chris Friesen
My previous testing with unix sockets prompted me to do a few lmbench runs with 2.4.19 and 2.5.65. The results have me a bit concerned, as there is no area where 2.5 is faster and several where it is significantly slower. In particular:
stat is 8 times worse
Quoted 8 lines
open/close are 7 times worse
fork is twice as expensive
tcp latency is 5 times worse
file deletion and mmap are both twice as expensive
tcp bandwidth is 5 times worse
Optimizing for muliple processors and heavy loads is nice, but this looks like its happening at the cost of basic performance. Is this really the route we should be taking?
These aren't terribly informative without profiles (esp. cache perfctrs).
TCP to localhost was explained to me as some excess checksumming that will eventually get removed before 2.6.0.
It's unclear why open()/close()/stat()/unlink() should be any different.
fork() is just rmap stuff. Try 2.5.65-mm2 and 2.5.65-mm3.
-- wli
Quoted 14 lines
My previous testing with unix sockets prompted me to do a few lmbench runs with 2.4.19 and 2.5.65. The results have me a bit concerned, as there is no area where 2.5 is faster and several where it is significantly slower.
In particular:
stat is 8 times worse
open/close are 7 times worse
fork is twice as expensive
tcp latency is 5 times worse
file deletion and mmap are both twice as expensive tcp bandwidth is 5 times worse
Quoted 3 lines
Optimizing for muliple processors and heavy loads is nice, but this looks like its happening at the cost of basic performance. Is this really the route we should be taking?
I think you're jumping to conclusions about what causes this - let's actually try to find the real root cause. These things have many different causes ... for instance, rmap has been found to be a problem in some workloads (especially things like the fork stuff). If you want to try 65-mjb1 with and without the the shared pagetable stuff, you may get some different results. (if you have stability problems, try doing a patch -p1 -R of 400-shpte, it seems a little fragile right now).
http://www.kernel.org/pub/linux/kernel/people/mbligh/2.5.65/
Also, if you can get kernel profiles for each test, that'd help to work out the root cause.
M.
In article <[email protected]>,
Quoted 4 lines · Chris Friesen
My previous testing with unix sockets prompted me to do a few lmbench runs with 2.4.19 and 2.5.65. The results have me a bit concerned, as there is no area where 2.5 is faster and several where it is significantly slower.
Try it with a modern library (like the one in the RH phoebe beta), and you'll see system calls having sped up by a factor of two. Even on UP.
But there's certainly something wrong with your open/close/stat numbers. I don't see anywhere _near_ those kinds of differences, and there are no real SMP locking issues there either. Are you sure you're testing the same setup?
Oh, and the TCP bandwidth thing is at least partly due to the fact that TCP loopback does extra copies due to debugging code being enabled.
Linus
Re: lmbench results for 2.4 and 2.5 -- updated results
In reply to Chris FriesenOkay, I'm somewhat chagrined but a bit relieved at the same time. Linus' comment about being sure that I'm testing the same setup promped me to go through and doublecheck my config. Turns out that I had some debug stuff turned on. Duh.
Here are the results of 2.4.20 and 2.5.65 with as close to matching configs as I could make them.
The ones that stand out are:
--fork/exec (due to rmap I assume?)
--mmap (also due to rmap?)
--select latency (any ideas?)
--udp latency (related to select latency?)
--page fault (is this significant?)
--tcp bandwidth (explained as debugging code)
Sorry about the bogus numbers last time around.
Chris
L M B E N C H 2 . 0 S U M M A R Y
------------------------------------
Processor, Processes - times in microseconds - smaller is better
----------------------------------------------------------------
Host OS Mhz null null open selct sig sig fork exec sh
call I/O stat clos TCP inst hndl proc proc proc
---- ------------- ---- ---- ---- ---- ---- ----- ---- ---- ---- ---- ----
doug Linux 2.5.65 750 0.38 0.73 5.46 7.13 64.2 1.03 3.25 231. 1729 17.K doug Linux 2.4.20 750 0.37 0.50 3.84 5.48 17.5 0.96 3.36 185. 1373 15.K
Context switching - times in microseconds - smaller is better
-------------------------------------------------------------
Host OS 2p/0K 2p/16K 2p/64K 8p/16K 8p/64K 16p/16K 16p/64K
ctxsw ctxsw ctxsw ctxsw ctxsw ctxsw ctxsw
---- ------------- ----- ------ ------ ------ ------ ------- -------
doug Linux 2.5.65 1.420 2.9700 108.7 46.6 157.6 46.7 157.5 doug Linux 2.4.20 1.120 2.3400 91.5 43.5 155.5 45.2 156.0
*Local* Communication latencies in microseconds - smaller is better
-------------------------------------------------------------------
Host OS 2p/0K Pipe AF UDP RPC/ TCP RPC/ TCP
ctxsw UNIX UDP TCP conn
---- ------------- ----- ----- ---- ----- ----- ----- ----- ----
doug Linux 2.5.65 1.420 7.642 11.4 21.9 45.0 27.2 60.5 104. doug Linux 2.4.20 1.120 6.606 10.3 15.8 40.9 26.2 56.5 82.9
File & VM system latencies in microseconds - smaller is better
--------------------------------------------------------------
Host OS 0K File 10K File Mmap Prot Page
Create Delete Create Delete Latency Fault Fault
---- ------------- ------ ------ ------ ------ ------- ----- -----
doug Linux 2.5.65 64.8 21.0 165.6 42.0 2550.0 0.946 4.00000
doug Linux 2.4.20 66.1 20.5 192.8 51.6 1612.0 0.764 2.00000*Local* Communication bandwidths in MB/s - bigger is better
-----------------------------------------------------------
Host OS Pipe AF TCP File Mmap Bcopy Bcopy Mem Mem
UNIX reread reread (libc) (hand) read write
---- ------------- ---- ---- ---- ------ ------ ------ ------ ---- -----
doug Linux 2.5.65 196. 107. 51.1 222.5 363.4 217.9 217.6 489. 326.0 doug Linux 2.4.20 233. 111. 90.0 253.6 370.0 223.8 226.1 498. 328.9
Signature · 4 lines
Chris Friesen | MailStop: 043/33/F10 Nortel Networks | work: (613) 765-0557 3500 Carling Avenue | fax: (613) 765-2986 Nepean, ON K2H 8E9 Canada | email: [email protected]
Re: lmbench results for 2.4 and 2.5 -- updated results
In article <[email protected]>,
Quoted 7 lines · Chris Friesen
Here are the results of 2.4.20 and 2.5.65 with as close to matching configs as I could make them.
The ones that stand out are:
--fork/exec (due to rmap I assume?)
--mmap (also due to rmap?)
Yes. You could try the objrmap patches, they are supposed to help. They may be in -mm, I'm not sure.
Quoted 1 line
--select latency (any ideas?)
I think this is due to the extra TCP debugging, but it might be something else. To disable the debugging, remove the setting of NETIF_F_TSO in linux/drivers/net/loopback.c, and re-test:
/* Current netfilter will die with oom linearizing large skbs,
* however this will be cured before 2.5.x is done.
*/
dev->features |= NETIF_F_TSO;Quoted 1 line
--udp latency (related to select latency?)
I doubt it. But there might be some more overhead somewhere. You should also run lmbench at least three times to get some feeling for the variance of the numbers, it can be quite big.
Quoted 1 line
--page fault (is this significant?)
I don't think so, there's something strange with the lmbench pagefault tests, it only has one significant digit of accuracy, and I don't even know what it is testing. Because of the single lack of precision, it's hard to tell what the real change is.
Quoted 1 line
--tcp bandwidth (explained as debugging code)
See if the NETIF_F_TSO change makes any difference. If performance is still bad, holler.
Linus
Re: lmbench results for 2.4 and 2.5 -- updated results
Quoted 3 lines · Chris Friesen
Quoted 3 lines
The ones that stand out are:
--fork/exec (due to rmap I assume?)
--mmap (also due to rmap?)
Quoted 2 lines · Linus Torvalds
Yes. You could try the objrmap patches, they are supposed to help. They may be in -mm, I'm not sure.
I recently asked Randy Hron which 2.5.x patches made the biggest difference in the tests he's done. He pasted the following:
<hrandoz:#kernelnewbies> null null
+ open signal signal fork execve /bin/sh
<hrandoz:#kernelnewbies> kernel call I/O
+stat fstat close install handle process process process
<hrandoz:#kernelnewbies> 2.5.65 0.66 0.96298 3
+.60 1.48 5.31 1.92 3.89 1279 3233 13703
<hrandoz:#kernelnewbies> 2.5.65-mm1 0.63 1.04114 3
+.65 1.57 6.39 2.29 3.92 1370 3621 13985
<hrandoz:#kernelnewbies> 2.5.65-mm2 0.65 0.98654 3
+.64 1.46 6.88 1.91 3.94 1511 3676 13502
<hrandoz:#kernelnewbies> 2.5.65-mm2-anobjrmap 0.66 0.96061 3
+.82 1.45 5.38 1.90 4.68 1414 3497 13169
<hrandoz:#kernelnewbies> 2.2.23 0.42 0.80455 4
+.76 1.24 5.77 1.43 2.74 788 2303 30829
<hrandoz:#kernelnewbies> 2.4.21-pre4aa3 0.62 0.72201 3
+.44 1.02 5.32 1.41 3.43 848 2114 10117
<hrandoz:#kernelnewbies> 2.4.21-pre5 0.62 0.75284 3
+.18 1.02 5.35 1.41 3.25 927 2559 11884
<hrandoz:#kernelnewbies> 2.4.21-pre5-akpm 0.61 0.73119 3
+.32 1.02 5.28 1.41 3.16 865 2421 11636
<hrandoz:#kernelnewbies> 2.5.63-mjb1 0.66 1.12795 4
+.01 1.64 6.66 1.92 4.49 1125 2793 12475
<hrandoz:#kernelnewbies> 2.5.62-mjb2 0.64 1.09703 4
+.12 1.66 5.77 1.89 4.05 1128 2888 12669
<hrandoz:#kernelnewbies> 2.5.63-mjb2 0.67 1.03824 4
+.12 1.66 5.87 1.90 4.39 1144 2985 12650
<hrandoz:#kernelnewbies> 2.5.62-mm3 0.62 0.95155 4
+.72 1.42 7.55 1.90 3.92 1164 3073 13101
-- wli