specweb99: dcache scalability results

At OLS, Hanna presented some of our work related to directory entry cache. I am presenting another set of interesting results that are a part of this ongoing work.

Mainly, we have been working on two tracks - reduce dcache_lock acquisitions by holding it while walking a cached path as suggested by Al Viro (fastwalk) and do a completely lockfree lookup/walk using RCU and lazy updation of the LRU list (dcache_rcu). Both showed promise and Troy Wilson from LTC perfromance team did a comparative study of these two dcache patches using specweb99. His measurement identifies what is good for dentry cache to use. Here is a summary of his reasults.

The measurements were done on a 8-CPU PIII Xeon server with more RAM than you and I can dream of getting in your desktop ;-)

More details of the directory entry cache work can be found in
http://lse.sourceforge.net/locking/dcache/dcache.html.
The dcache patches are available from Read-Copy Update package in http://www.sourceforge.net/projects/lse.

Throughput comparison:
---------------------

kernel throughput % improvement
(simultaneous connections)
----- -------------------------- -------------
2.4.17+lse02E 2258 -
2.4.17+lse02E+fastwalk 2280 1%
2.4.17+lse02E+dcache_rcu 2530 12%

Lockmeter comparison: (with apology for the > 80 col text)
---------------------

2.4.17+lse02E:

SPINLOCKS HOLD WAIT

  UTIL  CON    MEAN(  MAX )   MEAN(  MAX )(% CPU)     TOTAL NOWAIT SPIN RJECT  NAME
 15.7% 20.8%  2.1us(6668us)   23us(  14ms)( 4.4%)   5215460 79.2% 20.8% 0%  dcache_lock

2.4.17+lse02E+fastwalk:

SPINLOCKS HOLD WAIT

  UTIL  CON    MEAN(  MAX )   MEAN(  MAX )(% CPU)     TOTAL NOWAIT SPIN RJECT  NAME
 17.2% 17.7%  7.0us(  13ms)   53us(  30ms)( 2.9%)   1608566 82.3% 17.7% 0%  dcache_lock

2.4.17+lse02E+dcache_rcu:

SPINLOCKS HOLD WAIT

  UTIL  CON    MEAN(  MAX )   MEAN(  MAX )(% CPU)     TOTAL NOWAIT SPIN RJECT  NAME
  1.9%  2.3%  2.0us(3343us)   71us(9406us)(0.20%)    657152 97.7%  2.3% 0%  dcache_lock

Conclusions:
------------

Fastwalk clearly helps by reducing the number of dcache_lock acquisitions by 69.9%. However, holding the lock over entire walk of the path is clearly detrimental to performance as seen by the 3-fold increase in average hold time and average wait time. dcache_rcu has the biggest positive impact on performance for webserver type of workload. Its gains come from two improvements - 87.3% reduction in lock acquisitions at the same time keeping the lock hold time constant.

Acknowledgements:
-----------------

SPEC(tm) and the benchmark name SPECweb(tm) are registered trademarks of the Standard Performance Evaluation Corporation. The benchmarking were done for research purpose only and were non-compliant with the following devaitions from the rules -

1. It was run on hardware that does not meet the SPEC availability-to-the-public criteria. The machine was an engineering sample.
2. access_log wasn't kept for full accounting. It was being written, but deleted every 200 seconds.

For the latest SPECweb99 benchmark results visit http://www.spec.org

Thanks

Dipankar Sarma  <[email protected]> http://lse.sourceforge.net
Linux Technology Center, IBM Software Lab, Bangalore, India.