faster git clone

Hi William, this is a question much better directed at the Git list as a whole.

In general, it would be more compelling to see actual numbers than "much faster", e.g. the outputs of `time git clone
https://github.com/git/git and time aria2c
https://github.com/git/git/archive/master.zip` - or even an estimation from you, like, "I think clone takes a minute or two but aria does the same thing in only a couple of seconds". "Much faster" means something different to everyone :)

Right here it looks like your zip download redirects to a CDN or something, which is probably better optimized for serving archives than the Git server itself, so I would guess that has something to do with it too.

There are others on the list who are better able to explain this than me. But I'd guess the upshot is that 'git clone
https://github.com/git/git' is asking a Git server, which is good at Git repo management (e.g. accepting pushes, generating packfiles to send you a specific object or branch, etc) - but when you ask for "git/git/archive/master.zip" you're getting the result of some work the Git server already did a while ago to zip up the current 'master' into an archive and give it to some other server.

We've done some other work[1] around enabling use of CDNs and prebuilt chunks lately, but again, there are others on the list better able to explain than me.

[1]: https://github.com/git/git/blob/master/Documentation/technical/packfile-uri.txt

When Git shows the download speed, I believe it shows the speed at the given interval. So it may be that at the given moment, performance varies. Part of that is server side, since Git performs compression of data, and part of that is client side. For instance, if you're using SHA-1-DC (which you should, for security), there is a theoretical performance limit for clones of about 50 MiB/s on the client side which would be improved if you were cloning a SHA-256 repository[0].

If this is a fresh clone, it is probably already cached on the server side, since frequently requested pack files are cached at GitHub (although I'm not clear on whether a given request is cached can be determined), so it could be that it's just pushing data over the pipe as fast as your system can process it.

This is indeed backed by a CDN which may be much closer to you physically. Without seeing the full request, it's hard for me to say where your request was served from (CDN or not). I should point out that in this case you're cloning a single revision (so much less data), the data is usually cached, and your end of the system is not performing any decompression or hash verification, so it may appear faster when you're not performing equivalent work.

(I would kindly ask that you not try to download every revision in history for a comparison, because that would be a clearly excessive and abusive level of usage.)

I should also point out that you can't use multiple download threads to download from these endpoints because they don't, in general, handle Range requests. (Basically, they do if they're already cached, but not if they're not.)

It's impossible for me to say definitively what the performance problem is in this case, but I don't think it's intrinsically Git if you're seeing less than a 50 MiB/s speed. Git can and does process data at that speed on the local system (and at 15 MiB/s on my local network over Wi-Fi), so I'd guess that it's either a limitation in network performance based on two different serving locations or perhaps a temporarily overloaded server combined with the packfile not being cached.

[0] I personally think 50 MiB/s is a very reasonable transfer speed, but some people disagree.

brian m. carlson (he/him or they/them)
Houston, Texas, US