RFC Design for offloading part of packfile response to CDN

Some of us have been working on a design to improve the scalability of Git servers by allowing them to offload part of the packfile response to CDNs in this way: returning HTTP(S) URIs in fetch responses in addition to packfiles.

This can reduce the load on individual Git servers and improves proximity (by having data served from closer to the user).

I have included here a design document (patch 2) and a rough implementation of the server (patch 5). Currently, the implementation only allows replacing single blobs with URIs, but the protocol improvement is designed in such a way as to allow independent improvement of Git server implementations.

There is a potential issue: a server which produces both the URIs and the packfile at roughly the same time (like the implementation in this patch set) will not have sideband access until it has concluded sending the URIs. Among other things, this means that the server cannot send keepalive packets until quite late in the response. One solution to this might be to add a feature that allows the server to use a sideband throughout the whole response - and this has other benefits too like allowing servers to inform the client throughout the whole fetch, not just at the end.

Jonathan Tan (5):
Documentation: order protocol v2 sections
Documentation: add Packfile URIs design doc
upload-pack: refactor reading of pack-objects out
upload-pack: refactor writing of "packfile" line
upload-pack: send part of packfile response as uri

 Documentation/technical/packfile-uri.txt |  83 +++++++++++++
 Documentation/technical/protocol-v2.txt  |  22 ++--
 builtin/pack-objects.c                   |  48 ++++++++
 fetch-pack.c                             |   9 ++
 t/t5702-protocol-v2.sh                   |  25 ++++
 upload-pack.c                            | 150 ++++++++++++++++-------
 6 files changed, 285 insertions(+), 52 deletions(-)
 create mode 100644 Documentation/technical/packfile-uri.txt

2.19.0.271.gfe8321ec05.dirty