gitweb: Simple file based output caching

This 17+ patches long patch series is intended as a replacement for large 'gitweb: File based caching layer (from git.kernel.org)' mega-patch by John 'Warthog9' Hawley aka J.H., by starting small and adding features piece by piece.

This is fourthfifth version (5th release) of this series, and is available on http://repo.or.cz/w/git/jnareb-git.git as 'gitweb/cache-kernel-v5' branch. Earlier versions are available there as branches 'gitweb/cache-kernel', 'gitweb/cache-kernel-v2', 'gitweb/cache-kernel-v3' and 'gitweb/cache-kernel-v4'. Previous version of this series was sent to git mailing list as:

* [RFC PATCHv4 00/17] gitweb: Simple file based output caching
Message-Id: <[email protected]>
http://thread.gmane.org/gmane.comp.version-control.git/149101

This version is based on top of aad24ad (gitweb: Move call to evaluate_git_version after evaluate_gitweb_config, 2010-10-05) in 'gitweb/web' branch of http://repo.or.cz/w/git/jnareb-git.git
repository.

The 'gitweb/web' branch consist of the following commits on top of 1e63341 (Merge branch 'maint', 2010-09-30) in 'master' branch of main git repository:
gitweb/Makefile: Include gitweb/config.mak t/gitweb-lib.sh: Add support for GITWEB_TEST_INSTALLED gitweb/Makefile: Add 'test' and 'test-installed' targets
gitweb: Move call to evaluate_git_version after evaluate_gitweb_config

The first three patches were send as 3-part series:
* [PATCHv2 0/3] Testing installed gitweb
Message-Id: <[email protected]>
http://thread.gmane.org/gmane.comp.version-control.git/157222

Those are required to be able to test installed version of gitweb and modules, to check if there are any problems with process of installing gitweb.

The last patch in 'gitweb/web' branch was send as:
* [PATCH] gitweb: Move call to evaluate_git_version after evaluate_gitweb_config
Message-ID: <[email protected]>
http://article.gmane.org/gmane.comp.version-control.git/157220

This was needed during development of this series, to avoid spurious output of gitweb tests when using `--debug' option.

The MAIN DIFFERENCE from previous version is that it is based on never version of gitweb; in particular putting all code in subroutines: gitweb calls run() subroutine on startup, and run_request() subroutine on each request (per connection) from the 'jn/gitweb-fastcgi' branch.

The patches
gitweb: Return or exit after done serving request
gitweb: Fix typo in hash key name in %opts in git_header_html

present at beginning of previous (fourth) version of this series are already merged in.

The patches
gitweb/lib - Benchmarking GitwebCache::SimpleFileCache (in t/9603/) gitweb/lib - Alternate ways of capturing output

which were interleaved with other patches in previous version of this series got moved to the end of series, and marked proof of concept ("PoC").

The following changes are available in the git repository at:

git://repo.or.cz/git/jnareb-git.git gitweb/cache-kernel-v5

All comments welcome!

J.H., could you comment on how this series relates to the gitweb code *currently* running on git.kernel.org wrt. code? If possible, could you try to compare performance of those two implementations, the one presented here in this series, and the one used by git.kernel.org.

Pasky or other repo.or.cz admins, could you tell us if and what kind of caching gitweb used by repo.or.cz uses, and how it compares (wrt code and if possible also performance) to solution implemented in this series?

TODO list and areas of possible improvements would be send in separate email.

Jakub Narebski (21):
t/test-lib.sh: Export also GIT_BUILD_DIR in test_external
gitweb: Prepare for splitting gitweb
gitweb/lib - Very simple file based cache
gitweb/lib - Stat-based cache expiration
gitweb/lib - Regenerate entry if the cache file has size of 0 gitweb/lib - Simple select(FH) based output capture gitweb/lib - Cache captured output (using get/set)
gitweb: Add optional output caching (from gitweb/cache.pm) gitweb/lib - Adaptive cache expiration time
gitweb/lib - Use CHI compatibile (compute method) caching interface gitweb/lib - Use locking to avoid 'cache miss stampede' problem gitweb/lib - No need for File::Temp when locking gitweb/lib - Serve stale data when waiting for filling cache gitweb/lib - Regenerate (refresh) cache in background
gitweb: Introduce %actions_info, gathering information about actions
gitweb: Show appropriate "Generating..." page when regenerating cache
gitweb: Add startup delay to activity indicator for cache gitweb/lib - Add clear() and size() methods to caching interface
gitweb: Add beginnings of cache administration page gitweb/lib - Benchmarking GitwebCache::SimpleFileCache (in t/9603/) gitweb/lib - Alternate ways of capturing output

 gitweb/Makefile                                |   24 ++-
 gitweb/README                                  |   61 +++
 gitweb/gitweb.perl                             |  459 ++++++++++++++++++++++--
 gitweb/lib/GitwebCache/CacheOutput.pm          |   95 +++++
 gitweb/lib/GitwebCache/Capture.pm              |   66 ++++
 gitweb/lib/GitwebCache/Capture/PerlIO.pm       |   79 ++++
 gitweb/lib/GitwebCache/Capture/SelectFH.pm     |   82 ++++
 gitweb/lib/GitwebCache/Capture/TiedCapture.pm  |   76 ++++
 gitweb/lib/GitwebCache/FileCacheWithLocking.pm |  280 ++++++++++++++
 gitweb/lib/GitwebCache/SimpleFileCache.pm      |  472 ++++++++++++++++++++++++
 gitweb/lib/Tie/Restore.pm                      |   24 ++
 gitweb/lib/TiedCapture/PerlIO.pm               |   56 +++
 gitweb/lib/TiedCapture/String.pm               |   53 +++
 t/t9500-gitweb-standalone-no-errors.sh         |   20 +
 t/t9503-gitweb-caching-interface.sh            |   34 ++
 t/t9503/benchmark_caching_interface.pl         |  209 +++++++++++
 t/t9503/test_cache_interface.pl                |  407 ++++++++++++++++++++
 t/t9504-gitweb-capture-interface.sh            |   34 ++
 t/t9504/benchmark_capture_implementations.pl   |  226 +++++++++++
 t/t9504/test_capture_implementations.pl        |   85 +++++
 t/t9504/test_capture_interface.pl              |   76 ++++
 t/t9505-gitweb-cache.sh                        |   34 ++
 t/t9505/test_cache_output.pl                   |   70 ++++
 t/test-lib.sh                                  |    4 +-
 24 files changed, 2997 insertions(+), 29 deletions(-)
 create mode 100644 gitweb/lib/GitwebCache/CacheOutput.pm
 create mode 100644 gitweb/lib/GitwebCache/Capture.pm
 create mode 100644 gitweb/lib/GitwebCache/Capture/PerlIO.pm
 create mode 100644 gitweb/lib/GitwebCache/Capture/SelectFH.pm
 create mode 100644 gitweb/lib/GitwebCache/Capture/TiedCapture.pm
 create mode 100644 gitweb/lib/GitwebCache/FileCacheWithLocking.pm
 create mode 100644 gitweb/lib/GitwebCache/SimpleFileCache.pm
 create mode 100644 gitweb/lib/Tie/Restore.pm
 create mode 100644 gitweb/lib/TiedCapture/PerlIO.pm
 create mode 100644 gitweb/lib/TiedCapture/String.pm
 create mode 100755 t/t9503-gitweb-caching-interface.sh
 create mode 100755 t/t9503/benchmark_caching_interface.pl
 create mode 100755 t/t9503/test_cache_interface.pl
 create mode 100755 t/t9504-gitweb-capture-interface.sh
 create mode 100755 t/t9504/benchmark_capture_implementations.pl
 create mode 100755 t/t9504/test_capture_implementations.pl
 create mode 100755 t/t9504/test_capture_interface.pl
 create mode 100755 t/t9505-gitweb-cache.sh
 create mode 100755 t/t9505/test_cache_output.pl

1.7.3