Radix-tree pagecache for 2.5
Benchmark results on 2.4.17 and 2.4.17 with radix-tree (ratpagecache) patch. Identical results removed.
dbench 192 was the same. dbench 64 is virtually equal, considering dbench normal flucutation.
dbench 64 processes
2.4.17 ************************* 12.5 MB/sec
2.4.17rat ************************ 12.1 MB/sec
Unixbench-4.1.0
2.4.17 2.4.17rat
Pipe Throughput 387881.3 379702.9
Pipe-based Context Switching 105911.3 91653.3
Process Creation 1180.3 1197.4
System Call Overhead 304463.9 335158.8
Shell Scripts (1 concurrent) 617.4 615.6 C Compiler Throughput 225.7 227.7
Dc: sqrt(2) to 99 decimal places 13988.7 14360.6
LMbench 2.0p2
Processor, Processes - times in microseconds - smaller is better
----------------------------------------------------------------
OS null open selct sig sig fork exec sh
call stat clos TCP inst hndl proc proc proc
--------------- ---- ---- ---- ----- ---- ---- ----- ----- -----
Linux 2.4.17 0.43 3.41 6.16 36.8 1.43 3.26 932 3717 13612
Linux 2.4.17rat 0.43 4.56 7.80 55.4 1.45 3.17 901 3580 13239Context switching - times in microseconds - smaller is better
-------------------------------------------------------------
OS 2p/0K 2p/16K 2p/64K 8p/16K 8p/64K 16p/16K 16p/64K
ctxsw ctxsw ctxsw ctxsw ctxsw ctxsw ctxsw
---------------- ----- ------ ------ ------ ------ ------ ------
Linux 2.4.17 1.20 24.07 188.7 56.5 209.0 59.2 223.3
Linux 2.4.17rat 1.21 22.93 187.9 58.0 208.8 61.8 226.4*Local* Communication latencies in microseconds - smaller is better
-------------------------------------------------------------------
OS 2p/0K Pipe AF TCP TCP
ctxsw UNIX conn
--------------- ----- ----- ----- ----- -----
Linux 2.4.17 2.04 10.17 21.21 65.82 288.2
Linux 2.4.17rat 2.04 10.09 20.46 60.76 289.4File & VM system latencies in microseconds - smaller is better
--------------------------------------------------------------
OS 0K File 10K File Mmap Prot Page
Create Delete Create Delete Latency Fault Fault
--------------- ------ ------ ------ ------ ------- ----- -----
Linux 2.4.17 123.9 165.6 677.1 242.8 2602 1.076 8.0
Linux 2.4.17rat 128.9 169.7 618.8 256.2 2762 1.051 7.7*Local* Communication bandwidths in MB/s - bigger is better
-----------------------------------------------------------
OS Pipe AF TCP File Mmap Bcopy Bcopy Mem Mem
UNIX reread reread (libc) (hand) read write
--------------- ----- ----- ----- ------ ------ ------ ------ ----- -----
Linux 2.4.17 61.5 49.2 60.5 61.7 237.4 59.2 60.3 237.3 85.0
Linux 2.4.17rat 64.4 42.8 60.3 61.4 237.5 59.1 60.2 237.4 84.7Memory latencies in nanoseconds - smaller is better
---------------------------------------------------
OS Mhz L1 $ L2 $ Main mem ----------------- ---- ----- ------ -------- Linux 2.4.17 501 4.2 188.1 262.2 Linux 2.4.17rat 501 4.2 195.4 262.0
Threaded I/O Bench
Read, Write, and Seeks are MB/sec
CPU Effiency (CPU Eff) = (MB/sec) / CPU% (bigger is better)
Num Seq Read CPU Rand Read CPU Thr Rate (CPU%) Eff Rate (CPU%) Eff --- ------------------- ----------------- 2.4.17 1 13.23 43.2% 30.62 2.74 3.7% 73.88 2.4.17rat 1 12.60 43.0% 29.30 2.75 3.8% 72.45
2.4.17 2 11.56 29.9% 38.66 3.04 3.8% 79.89 2.4.17rat 2 11.07 30.4% 36.41 3.06 4.3% 71.81
2.4.17 4 11.03 25.9% 42.59 3.19 4.1% 78.26 2.4.17rat 4 10.62 26.5% 40.08 3.15 4.2% 74.32
2.4.17 8 10.62 22.8% 46.58 3.29 4.2% 77.99 2.4.17rat 8 10.23 23.4% 43.72 3.26 4.5% 73.05
Num Seq Write CPU Rand Write CPU Thr Rate (CPU%) Eff Rate (CPU%) Eff --- ------------------- ----------------- 2.4.17 1 11.08 50.5% 21.94 0.69 1.6% 44.10 2.4.17rat 1 7.77 32.8% 23.69 0.53 1.1% 48.44
2.4.17 2 10.83 48.6% 22.28 0.69 1.5% 45.13 2.4.17rat 2 7.51 32.1% 23.38 0.52 1.1% 45.98
2.4.17 4 10.40 45.9% 22.66 0.68 1.5% 44.70 2.4.17rat 4 7.62 32.8% 23.25 0.53 1.2% 45.21
2.4.17 8 10.17 44.8% 22.70 0.67 1.5% 44.73 2.4.17rat 8 7.55 32.5% 23.24 0.53 1.2% 44.66
bonnie++
Version 1.02a ---------------------Sequential Output--------------------
-----Per Char----- ------Block------- -----Rewrite------ Kernel Size MB/sec %CPU Eff MB/sec %CPU Eff MB/sec %CPU Eff 2.4.17 1024 3.41 98.0 3.48 14.56 65.6 22.19 8.83 51.0 17.32 2.4.17rat 1024 3.36 98.0 3.43 10.64 41.6 25.57 6.79 37.2 18.27
Version 1.02a -----------Sequential Input----------- ------Random-----
-----Per Char----- ------Block------- ------Seeks------ Kernel Size MB/sec %CPU Eff MB/sec %CPU Eff /sec %CPU Eff 2.4.17 1024 3.98 97.2 4.10 16.39 60.6 27.05 132 2.0 6609 2.4.17rat 1024 3.92 96.0 4.08 15.46 57.0 27.12 126 2.0 6302
-------Sequential Create-----------
------Create----- -----Delete-----
files /sec %CPU Eff /sec %CPU Eff
2.4.17 16384 4421 96.8 4567 4719 97.4 4845
2.4.17rat 16384 3973 97.6 4070 4531 95.0 4769 -------Random Create----------------
------Create----- -----Delete-----
files /sec %CPU Eff /sec %CPU Eff
2.4.17 16384 4475 98.0 4566 4124 94.0 4387
2.4.17rat 16384 4203 98.0 4288 4005 94.4 4242Build times. Smaller is better - task completed faster.
perl_build kernel_build
2.4.17 1620 1347
2.4.17rat 1629 1444
radix-tree saves 768k on 384M machine.
2.4.17rat Memory: 385932k/393216k available (885k kernel code, 6900k reserved
2.4.17 Memory: 385164k/393216k available (884k kernel code, 7668k reserved
Results like these on more kernels at:
http://home.earthlink.net/~rwhron/kernel/k6-2-475.html
Signature · 1 line
Randy Hron
Quoted 5 lines · On Sat, 2 Feb 2002 [email protected] wrote:
(quoted diff) +0
-0Num Seq Write CPU Rand Write CPU Thr Rate (CPU%) Eff Rate (CPU%) Eff --- ------------------- -----------------2.4.17 1 11.08 50.5% 21.94 0.69 1.6% 44.10 2.4.17rat 1 7.77 32.8% 23.69 0.53 1.1% 48.44
This is a worrying trend your benching has exposed - all the streaming write tests have taken a performance hit. Above is tiobench, and bonnie showed the same trend too.
Chris
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"rwhron" == rwhron <[email protected]> writes:
rwhron> bonnie++
rwhron> Version 1.02a ---------------------Sequential Output-------------------- rwhron> -----Per Char----- ------Block------- -----Rewrite------ rwhron> Kernel Size MB/sec %CPU Eff MB/sec %CPU Eff MB/sec %CPU Eff rwhron> 2.4.17 1024 3.41 98.0 3.48 14.56 65.6 22.19 8.83 51.0 17.32 rwhron> 2.4.17rat 1024 3.36 98.0 3.43 10.64 41.6 25.57 6.79 37.2 18.27
rwhron> Version 1.02a -----------Sequential Input----------- ------Random----- rwhron> -----Per Char----- ------Block------- ------Seeks------ rwhron> Kernel Size MB/sec %CPU Eff MB/sec %CPU Eff /sec %CPU Eff rwhron> 2.4.17 1024 3.98 97.2 4.10 16.39 60.6 27.05 132 2.0 6609 rwhron> 2.4.17rat 1024 3.92 96.0 4.08 15.46 57.0 27.12 126 2.0 6302
rwhron> -------Sequential Create-----------
rwhron> ------Create----- -----Delete----- rwhron> files /sec %CPU Eff /sec %CPU Eff rwhron> 2.4.17 16384 4421 96.8 4567 4719 97.4 4845 rwhron> 2.4.17rat 16384 3973 97.6 4070 4531 95.0 4769
rwhron> -------Random Create----------------
rwhron> ------Create----- -----Delete----- rwhron> files /sec %CPU Eff /sec %CPU Eff rwhron> 2.4.17 16384 4475 98.0 4566 4124 94.0 4387 rwhron> 2.4.17rat 16384 4203 98.0 4288 4005 94.4 4242
Hmm, I've got different results with bonnie++, are you sure you didn't swap the results :)
Linux 2.5.3-dj1
--------------
Version 1.02b ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
merlin 496M 22038 13 11768 8 19219 6 183.9 0
merlin 496M 22495 13 11216 6 22390 6 183.9 0
merlin 496M 22292 13 11713 7 19249 6 188.8 0
------Sequential Create------ --------Random Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
merlin 10 2015 99 +++++ +++ +++++ +++ 2526 99 +++++ +++ 8514 99
merlin 10 2354 99 +++++ +++ +++++ +++ 2656 99 +++++ +++ 10195 100
merlin 10 2871 99 +++++ +++ +++++ +++ 3036 99 +++++ +++ 12012 98Linux 2.5.3-dj1-radix-pagecache
-------------------------------
Version 1.02b ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
merlin 496M 22397 13 11248 7 22595 7 178.6 0
merlin 496M 22088 12 11204 7 22591 7 188.1 0
merlin 496M 24767 14 10966 7 22474 7 191.1 0
------Sequential Create------ --------Random Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
merlin 10 2485 99 +++++ +++ +++++ +++ 2456 99 +++++ +++ 9614 100
merlin 10 2653 99 +++++ +++ +++++ +++ 2884 100 +++++ +++ 11101 100
merlin 10 1948 99 +++++ +++ +++++ +++ 2530 99 +++++ +++ 9294 99Quoted 2 lines · Momchil Velikov
Hmm, I've got different results with bonnie++, are you sure you didn't swap the results :)
I don't think so, but I notice that bonnie++ Sequential and Random Create tests flucuate a lot. These are on 16384 small files in my tests.
------Sequential Create------ --------Random Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
2.4.17 16 4217 97 +++++ +++ 4664 99 4354 99 +++++ +++ 4090 95
16 4423 99 +++++ +++ 4266 90 4404 95 +++++ +++ 4382 100
16 4468 97 +++++ +++ 4899 99 4522 99 +++++ +++ 4235 95
16 4498 96 +++++ +++ 4777 100 4464 99 +++++ +++ 3854 90
16 4503 95 +++++ +++ 4990 99 4632 98 +++++ +++ 4058 902.4.17rat 16 2994 98 +++++ +++ 4548 94 2952 99 +++++ +++ 3967 92
16 3055 97 +++++ +++ 4705 93 4665 99 +++++ +++ 4119 94
16 4463 96 +++++ +++ 4670 100 4452 100 +++++ +++ 3775 94
16 4833 99 +++++ +++ 4823 100 4616 97 +++++ +++ 4054 92
16 4521 98 +++++ +++ 3907 88 4329 95 +++++ +++ 4111 100The test against a 1GB (large) file vary somewhat too, but not as much as the tests above. The results I posted earlier were the averages from 5 runs.
I just uploaded the raw logfiles from my testing (not just 2.4.17 and radix-tree) to http://home.earthlink.net/~rwhron/kernel (the tar.bz2 files are the logs).
Signature · 1 line
Randy Hron
Quoted 2 lines
Hmm, I've got different results with bonnie++, are you sure you didn't swap the results :)
I am curious if the small followup patch to 2.5.3-dj1 makes radix-tree more like 2.4.17-ratpagecache. (overall, it seems that radix-tree did better in I/O without the small followup patch).
Benchmarks on 2.5.3-dj1, 2.5.3-dj1 with radix-tree (rat) and small followup patch (rat2).
dbench 64 processes
2.5.3-dj1rat ************************** 13.1 MB/sec 2.5.3-dj1 ********************** 11.1 MB/sec 2.5.3-dj1rat2 ********************** 11.1 MB/sec
dbench 192 processes
2.5.3-dj1rat ************* 6.8 MB/sec 2.5.3-dj1rat2 ************* 6.7 MB/sec 2.5.3-dj1 ************* 6.6 MB/sec
Unixbench-4.1.0
2.5.3-dj1 2.5.3-dj1rat 2.5.3-dj1rat2
Execl Throughput 316.7 298.5 324.8 lps
Pipe Throughput 307626.1 308399.0 284619.1 lps
Pipe-based Context Switching 95502.3 93398.1 76470.1 lps
Process Creation 1320.7 1280.5 1398.3 lps
System Call Overhead 245040.7 239410.7 251300.1 lps
Shell Scripts (1 concurrent) 643.2 639.0 636.1 lpm
Shell Scripts (8 concurrent) 89.3 88.0 86.9 lpm Shell Scripts (16 concurrent) 44.7 44.0 44.0 lpm Shell Scripts (32 concurrent) 22.7 22.0 22.0 lpm Shell Scripts (64 concurrent) 10.9 10.7 10.7 lpm Shell Scripts (96 concurrent) 7.1 7.1 6.9 lpm Shell Scripts (128 concurrent) 5.3 5.1 5.1 lpm C Compiler Throughput 220.6 225.2 217.1 lpm Dc: sqrt(2) to 99 decimal places 15838.5 15465.6 15854.2 lpm Recursion Test--Tower of Hanoi 14157.1 14156.8 14156.2 lps
LMbench-2.0p2 average of 9 samples.
Processor, Processes - times in microseconds - smaller is better
OS null open selct sig sig fork exec sh
call stat clos TCP inst hndl proc proc proc
-------------------- ---- ---- ---- ----- ---- ---- ---- ---- -----
Linux 2.5.3-dj1 0.60 3.56 5.42 39.0 1.49 3.10 777 3250 12082
Linux 2.5.3-dj1rat 0.60 4.65 6.61 37.5 1.44 3.03 809 3294 12580
Linux 2.5.3-dj1rat2 0.59 4.52 7.82 59.5 1.42 3.23 788 3431 12943Context switching - times in microseconds - smaller is better
OS 2p/0K 2p/16K 2p/64K 8p/16K 8p/64K 16p/16K 16p/64K
ctxsw ctxsw ctxsw ctxsw ctxsw ctxsw ctxsw
------------------- ----- ------ ------ ------ ------ ------ ------
Linux 2.5.3-dj1 1.04 18.98 190.4 55.5 207.3 59.0 224.3
Linux 2.5.3-dj1rat 0.43 19.57 183.7 56.6 204.0 58.1 225.0
Linux 2.5.3-dj1rat2 1.02 20.39 184.9 58.0 212.3 59.5 227.1*Local* Communication latencies in microseconds - smaller is better
OS Pipe AF TCP TCP
UNIX conn
------------------- ----- ----- ----- -----
Linux 2.5.3-dj1 13.57 27.21 76.68 303.6
Linux 2.5.3-dj1rat 10.46 20.51 72.37 292.6
Linux 2.5.3-dj1rat2 11.41 20.04 75.18 304.6File & VM system latencies in microseconds - smaller is better
OS 0K File 10K File Mmap Prot Page
Create Delete Create Delete Latency Fault Fault
------------------- ------ ------ ------ ------ ------- ----- -----
Linux 2.5.3-dj1 132.3 195.5 709.7 285.9 2821 1.134 6.1
Linux 2.5.3-dj1rat 145.5 198.2 791.6 291.7 2699 1.495 5.6
Linux 2.5.3-dj1rat2 140.1 195.0 723.2 287.4 2792 1.588 5.6*Local* Communication bandwidths in MB/s - bigger is better
OS Pipe AF TCP File Mmap Bcopy Bcopy Mem Mem
UNIX reread reread (libc) (hand) read write
------------------- ---- ---- ---- ------ ------ ------ ------ ----- ----- Linux 2.5.3-dj1 65.7 44.3 42.7 60.3 237.5 59.2 60.4 237.4 85.1 Linux 2.5.3-dj1rat 64.2 45.5 38.4 59.8 237.7 59.5 60.6 237.6 85.7 Linux 2.5.3-dj1rat2 67.0 46.0 45.3 60.1 237.6 59.4 60.5 237.4 85.4
Memory latencies in nanoseconds - smaller is better
OS Mhz L1 $ L2 $ Main mem
------------------- ---- ----- ------ -------- Linux 2.5.3-dj1 501 4.2 195.7 262.3 Linux 2.5.3-dj1rat 501 4.2 191.1 261.9 Linux 2.5.3-dj1rat2 501 4.2 189.9 262.1
Threaded I/O Bench
File Size is 384MB, Read, Write, and Seeks are MB/sec CPU Effiency (CPU Eff) = (MB/sec) / CPU% (bigger is better)
Reads Num Seq Read CPU Rand Read CPU
Thr Rate (CPU%) Eff Rate (CPU%) Eff --- ------------------- ----------------- 2.5.3-dj1 1 12.92 42.7% 30.26 3.21 4.7% 68.63 2.5.3-dj1rat 1 12.99 44.8% 29.00 2.80 3.9% 71.17 2.5.3-dj1rat2 1 12.75 43.2% 29.51 2.75 4.6% 59.29
2.5.3-dj1 2 11.63 32.1% 36.23 3.34 4.4% 75.57 2.5.3-dj1rat 2 11.70 33.3% 35.14 3.00 4.5% 65.93 2.5.3-dj1rat2 2 11.42 32.3% 35.36 2.92 4.9% 59.75
2.5.3-dj1 4 11.26 28.8% 39.10 3.42 4.8% 71.70 2.5.3-dj1rat 4 11.22 30.0% 37.40 3.06 4.7% 65.18 2.5.3-dj1rat2 4 11.14 29.5% 37.76 3.00 4.9% 61.69
2.5.3-dj1 8 11.25 26.8% 41.98 3.61 5.1% 71.02 2.5.3-dj1rat 8 10.96 26.9% 40.74 3.19 4.6% 69.09 2.5.3-dj1rat2 8 10.94 26.9% 40.67 3.09 4.9% 62.91
Writes Num Seq Write CPU Rand Write CPU Thr Rate (CPU%) Eff Rate (CPU%) Eff --- ------------------- ----------------- 2.5.3-dj1 1 11.38 55.9% 20.36 0.76 2.0% 37.21 2.5.3-dj1rat 1 10.77 54.3% 19.83 0.70 2.0% 35.30 2.5.3-dj1rat2 1 9.01 42.5% 21.21 0.57 1.5% 37.75
2.5.3-dj1 2 11.35 56.4% 20.12 0.77 2.0% 37.32 2.5.3-dj1rat 2 10.54 53.2% 19.81 0.70 2.0% 35.69 2.5.3-dj1rat2 2 7.92 37.0% 21.40 0.52 1.4% 38.15
2.5.3-dj1 4 11.39 57.3% 19.88 0.78 2.1% 36.65 2.5.3-dj1rat 4 10.54 53.2% 19.81 0.71 2.0% 35.91 2.5.3-dj1rat2 4 7.71 36.1% 21.37 0.51 1.4% 37.56
2.5.3-dj1 8 11.28 56.6% 19.93 0.79 2.2% 36.22 2.5.3-dj1rat 8 10.77 54.8% 19.65 0.73 2.0% 36.24 2.5.3-dj1rat2 8 7.93 37.3% 21.27 0.52 1.4% 37.16
bonnie++-1.02a summary
1024 MB file ---------------------Sequential Output--------------------
-----Per Char----- ------Block------- -----Rewrite------ Kernel MB/sec %CPU Eff MB/sec %CPU Eff MB/sec %CPU Eff 2.5.3-dj1 3.38 98.0 3.44 14.20 67.4 21.06 8.52 51.0 16.71 2.5.3-dj1rat 3.35 98.0 3.42 14.01 62.0 22.60 8.19 46.4 17.66 2.5.3-dj1rat2 3.38 98.0 3.44 13.06 56.2 23.24 7.66 42.6 17.98
-----------Sequential Input----------- ------Random-----
-----Per Char----- ------Block------- ------Seeks------
Kernel MB/sec %CPU Eff MB/sec %CPU Eff /sec %CPU Eff
2.5.3-dj1 3.96 97.4 4.07 15.26 57.2 26.68 129 2.0 6460
2.5.3-dj1rat 3.94 97.0 4.06 15.21 57.4 26.49 127 1.8 7051
2.5.3-dj1rat2 3.93 97.0 4.05 15.13 57.4 26.36 128 2.0 6415bonnie++ small files (16384 files)
--------Sequential Create------------
------Create----- -----Delete-----
/sec %CPU Eff /sec %CPU Eff
2.5.3-dj1 4369 97.4 4485 4176 96.8 4313
2.5.3-dj1rat 4276 98.0 4364 4100 96.0 4271
2.5.3-dj1rat2 4293 97.0 4425 3880 94.4 4110 ------------Random Create------------
------Create----- -----Delete-----
/sec %CPU Eff /sec %CPU Eff
2.5.3-dj1 4334 98.2 4413 3782 96.4 3923
2.5.3-dj1rat 4238 97.6 4341 3696 95.0 3890
2.5.3-dj1rat2 4215 97.2 4336 3667 95.0 3860Time to build/test/run in seconds
version autoconf perl kernel updatedb updatedb*5 2.5.3-dj1 1152 1636 1320 29 38 2.5.3-dj1rat 1167 1684 1336 29 39 2.5.3-dj1rat2 1162 1674 1356 29 37
Memory Info - radix-tree saved 768k on 384MB machine.
version available totalmem kern_code reserved
2.5.3-dj1rat 385860k 393216k 931k 6972k 2.5.3-dj1rat2 385860k 393216k 931k 6972k 2.5.3-dj1 385092k 393216k 930k 7740k
Details on system, testing method, and other results at:
http://home.earthlink.net/~rwhron/kernel/k6-2-475.html
Signature · 1 line
Randy Hron
Quoted 3 lines · On Tue, 5 Feb 2002 [email protected] wrote:
I am curious if the small followup patch to 2.5.3-dj1 makes radix-tree more like 2.4.17-ratpagecache. (overall, it seems that radix-tree did better in I/O without the small followup patch).
It would be useful if you also did dbench tests with a much lower amount of dbench processes.
Once you get over 'dbench 16' or so the whole thing basically becomes an excercise in how well the system can trigger task starvation in get_request_wait.
I can recommend 'dbench 1' 'dbench 4' 'dbench 16' ;)
Quoted 9 lines
dbench 64 processes
2.5.3-dj1rat ************************** 13.1 MB/sec 2.5.3-dj1 ********************** 11.1 MB/sec 2.5.3-dj1rat2 ********************** 11.1 MB/secdbench 192 processes
2.5.3-dj1rat ************* 6.8 MB/sec 2.5.3-dj1rat2 ************* 6.7 MB/sec 2.5.3-dj1 ************* 6.6 MB/sec
Rik
Signature · 4 lines
"Linux holds advantages over the single-vendor commercial OS"
-- Microsoft's "Competing with Linux" document
http://www.surriel.com/ http://distro.conectiva.com/