Smart fetch via HTTP?
Hello,
Did anyone already think about fetching over HTTP working similarly to the native git protocol?
That is rather than reading the raw content of the repository, there would be a CGI script (could be integrated to gitweb), that would negotiate what the client needs and then generate and send a single pack with it.
Mercurial and bzr both have this option. It would IMO have three benefits:
- Fast access for people behind paranoid firewalls, that only let http and https (you can tunel anything through, but only to port 443) through.
- Can be run on shared machine. If you have web space on machine shared by many people, you can set up your own gitweb, but cannot/are not allowed to start your own network server for git native protocol.
- Less things to set up. If you are setting up gitweb anyway, you'd not need to set up additional thing for providing fetch access.
Than a question is how to implement it. The current protocol is stateful on both sides, but the stateless nature of HTTP more or less requires the protocol to be stateless on the server.
I think it would be possible to use basically the same protocol as now, but make it stateless for server. That is server first sends it's heads and than client repeatedly sends all it's wants and some haves until the server acks all of them and sends the pack.
Alternatively I am thinking about using Bloom filters (somebody came with such idea on the bzr list when I still followed it). It might be useful, as over HTTP we need to send as many haves as possible in one go.
Signature · 1 line
Jan 'Bulb' Hudec <[email protected]>
Quoted 2 lines · Jan Hudec
Did anyone already think about fetching over HTTP working similarly to the native git protocol?
No work has been done on this (that I know of) but I've discussed it to some extent with Simon 'corecode' Schubert on #git, and I think he also brought it up on the mailing list not too long after.
I've certainly thought about adding some sort of pack-objects frontend into gitweb.cgi for this exact purpose. It is really quite easy, except for the negotation of what the client has. ;-)
Quoted 8 lines
Than a question is how to implement it. The current protocol is stateful on both sides, but the stateless nature of HTTP more or less requires the protocol to be stateless on the server.
I think it would be possible to use basically the same protocol as now, but make it stateless for server. That is server first sends it's heads and than client repeatedly sends all it's wants and some haves until the server acks all of them and sends the pack.
I think Simon was talking about doubling the number of haves the client sends in each request. So the client POSTs initially all of its current refs; then current refs and their parents; then 4 commits back, then 8, etc. The server replies to each POST request with either a "send more please" or the packfile.
Signature · 1 line
Shawn.
Quoted 21 lines · Shawn O. Pearce
Quoted 2 lines · Jan Hudec
Did anyone already think about fetching over HTTP working similarly to the native git protocol?
No work has been done on this (that I know of) but I've discussed it to some extent with Simon 'corecode' Schubert on #git, and I think he also brought it up on the mailing list not too long after.
I've certainly thought about adding some sort of pack-objects frontend into gitweb.cgi for this exact purpose. It is really quite easy, except for the negotation of what the client has. ;-)
Quoted 8 lines
Than a question is how to implement it. The current protocol is stateful on both sides, but the stateless nature of HTTP more or less requires the protocol to be stateless on the server.
I think it would be possible to use basically the same protocol as now, but make it stateless for server. That is server first sends it's heads and than client repeatedly sends all it's wants and some haves until the server acks all of them and sends the pack.
Quoted 5 lines
I think Simon was talking about doubling the number of haves the client sends in each request. So the client POSTs initially all of its current refs; then current refs and their parents; then 4 commits back, then 8, etc. The server replies to each POST request with either a "send more please" or the packfile.
I kinda' like the bundle suggestion ;-)
Hi,
Quoted 4 lines · Jan Hudec
Did anyone already think about fetching over HTTP working similarly to the That is rather than reading the raw content of the repository, there would be a CGI script (could be integrated to gitweb), that would negotiate what the client needs and then generate and send a single pack with it.
frankly, I'm not that excited. I'm not disputing that this would be
useful, but I have my doubts on just how *much* useful it would be - I'm not so sure the set of users affected is really all that large. So I'm just cooling people down here. ;-))
Quoted 3 lines
Mercurial and bzr both have this option. It would IMO have three benefits:
- Fast access for people behind paranoid firewalls, that only let http and https (you can tunel anything through, but only to port 443) through.
How many users really have this problem? I'm not so sure. There are
certainly some, but enough for this to be a viable argument?
Quoted 3 lines
- Can be run on shared machine. If you have web space on machine shared by many people, you can set up your own gitweb, but cannot/are not allowed to start your own network server for git native protocol.
You need to have CGI-enabled hosting, set up the CGI script etc. -
overally, the setup is similarly complicated as git-daemon setup, so it's not "zero-setup" solution anymore.
Again, I'm not sure just how many people are in the situation that
they can run real CGI (not just PHP) but not git-daemon.
Quoted 2 lines
- Less things to set up. If you are setting up gitweb anyway, you'd not need to set up additional thing for providing fetch access.
Except, well, how do you "set it up"? You need to make sure
git-update-server-info is run, yes, but that shouldn't be a problem (I'm not so sure if git does this for you automagically - Cogito would...).
I think 95% of people don't set up gitweb.cgi either for their small
HTTP repositories. :-)
Then again, it's not that it would be really technically complicated -
adding "give me a bundle" support to gitweb should be pretty easy. However, this support has some "social" costs as well: no compatibility with older git versions, support cost, confusion between dumb HTTP and gitweb HTTP transports, more lack of motivation for improving dumb HTTP transport...
Signature · 4 lines
Petr "Pasky" Baudis Stuff: http://pasky.or.cz/ Ever try. Ever fail. No matter. // Try again. Fail again. Fail better. -- Samuel Beckett
Quoted 11 lines · Petr Baudis
Quoted 3 lines · Jan Hudec
- Can be run on shared machine. If you have web space on machine shared by many people, you can set up your own gitweb, but cannot/are not allowed to start your own network server for git native protocol.
You need to have CGI-enabled hosting, set up the CGI script etc. -
overally, the setup is similarly complicated as git-daemon setup, so it's not "zero-setup" solution anymore.Again, I'm not sure just how many people are in the situation that
they can run real CGI (not just PHP) but not git-daemon.
A particular case would be a group of students wanting to publish their software project (I mean the PRG023 or equivalent). Private computers in the hostel are not allowed to serve anything, so they'd use some of the lab servers (eg. artax, ss1000...). All of them allow full CGI, but running daemons is forbiden.
Quoted 6 lines
Quoted 2 lines
- Less things to set up. If you are setting up gitweb anyway, you'd not need to set up additional thing for providing fetch access.
Except, well, how do you "set it up"? You need to make sure
git-update-server-info is run, yes, but that shouldn't be a problem (I'm not so sure if git does this for you automagically - Cogito would...).
No. If it worked similar to git-upload-pack, only over http, it would work without update-server-info, no?
Quoted 9 lines
I think 95% of people don't set up gitweb.cgi either for their small
HTTP repositories. :-)Then again, it's not that it would be really technically complicated -
adding "give me a bundle" support to gitweb should be pretty easy. However, this support has some "social" costs as well: no compatibility with older git versions, support cost, confusion between dumb HTTP and gitweb HTTP transports, more lack of motivation for improving dumb HTTP transport...
The dumb transport is definitely useful. Extending it to use ranges if possible would be useful as well (and maybe more than upload-pack-over-http).
Signature · 1 line
Jan 'Bulb' Hudec <[email protected]>
Quoted 5 lines · Jan Hudec
A particular case would be a group of students wanting to publish their software project (I mean the PRG023 or equivalent). Private computers in the hostel are not allowed to serve anything, so they'd use some of the lab servers (eg. artax, ss1000...). All of them allow full CGI, but running daemons is forbiden.
And wouldn't the admin authority for those lab servers be amenable to install a Git daemon service? That'd be a much better solution to me.
Nicolas
Quoted 10 lines · Nicolas Pitre
Quoted 5 lines · Jan Hudec
A particular case would be a group of students wanting to publish their software project (I mean the PRG023 or equivalent). Private computers in the hostel are not allowed to serve anything, so they'd use some of the lab servers (eg. artax, ss1000...). All of them allow full CGI, but running daemons is forbiden.
And wouldn't the admin authority for those lab servers be amenable to install a Git daemon service? That'd be a much better solution to me.
It would. But it would really depend on the administrator goodwill.
Signature · 1 line
Jan 'Bulb' Hudec <[email protected]>
Quoted 5 lines · Petr Baudis
Quoted 3 lines
Mercurial and bzr both have this option. It would IMO have three benefits:
- Fast access for people behind paranoid firewalls, that only let http and https (you can tunel anything through, but only to port 443) through.How many users really have this problem? I'm not so sure.
Many (if not most?) of the people working in a big company, I'd say. Year, it sucks, but people having used a paranoid firewall with a not-less-paranoid and broken proxy understand what I mean.
Quoted 10 lines
Quoted 3 lines
- Can be run on shared machine. If you have web space on machine shared by many people, you can set up your own gitweb, but cannot/are not allowed to start your own network server for git native protocol.
You need to have CGI-enabled hosting, set up the CGI script etc. -
overally, the setup is similarly complicated as git-daemon setup, so it's not "zero-setup" solution anymore.Again, I'm not sure just how many people are in the situation that
they can run real CGI (not just PHP) but not git-daemon.
Any volunteer to write a full-PHP version of git? ;-)
Signature · 1 line
Matthieu
Quoted 4 lines · Matthieu Moy
Many (if not most?) of the people working in a big company, I'd say. Year, it sucks, but people having used a paranoid firewall with a not-less-paranoid and broken proxy understand what I mean.
Well, we could try to support the git protocol over port 80..
IOW, it's probably easier to try to get people to use
git clone git://some.host:80/project
and just run git-daemon on port 80, than it is to try to set of magic cgi scripts etc.
Doing that with virtual hosts etc should be pretty trivial. Much more so than trying to make a git-cgi script.
And yes, I do realize that in theory you can have http-aware firewalls that expect to see the normal http sequences in the first few packets in order to pass things through, but I seriously doubt it's very common.
Linus
Quoted 8 lines · Linus Torvalds
Well, we could try to support the git protocol over port 80..
IOW, it's probably easier to try to get people to use
git clone
git://some.host:80/projectand just run git-daemon on port 80, than it is to try to set of magic cgi scripts etc.
Can we tech the git-daemon to parse the HTTP headers
(specifically, the URL) and return the appropriate HTTP response?
Quoted 3 lines
And yes, I do realize that in theory you can have http-aware firewalls that expect to see the normal http sequences in the first few packets in order to pass things through, but I seriously doubt it's very common.
It's not about packet scanning, it's about GET vs CONNECT. If
the proxy allows GET but not CONNECT, it's going to forward the HTTP protocol to the server, and git-daemon is going to see "GET /project HTTP/1.1" as its first input. Now, perhaps we can cook that up behind some apache so that apache handles vhosting the URL, then calls git-daemon which can take the stdin. So we'd be doing POST, not GET.
On the other hand, if the proxy allows CONNECT, there is no
scanning for HTTP sequences done by the proxy. It just allows all raw data (as it figures you're doing SSL).
A normal company needs to have their firewall allow CONNECT to
9418. Then git proxying over HTTP is possible to a standard git-daemon.
Joel
Signature · 9 lines
"The first requisite of a good citizen in this republic of ours is that he shall be able and willing to pull his weight." - Theodore Roosevelt Joel Becker Principal Software Developer Oracle E-mail: [email protected] Phone: (650) 506-8127
Quoted 7 lines · Joel Becker
It's not about packet scanning, it's about GET vs CONNECT. If
the proxy allows GET but not CONNECT, it's going to forward the HTTP protocol to the server, and git-daemon is going to see "GET /project HTTP/1.1" as its first input. Now, perhaps we can cook that up behind some apache so that apache handles vhosting the URL, then calls git-daemon which can take the stdin. So we'd be doing POST, not GET.
If it's _just_ the initial GET/CONNECT strings, yeah, we could probably easily make the git-daemon just ignore them. That shouldn't be a problem.
But if there's anything *else* required, it gets uglier much more quickly.
Linus
Quoted 4 lines · Linus Torvalds
If it's _just_ the initial GET/CONNECT strings, yeah, we could probably easily make the git-daemon just ignore them. That shouldn't be a problem.
But if there's anything *else* required, it gets uglier much more quickly.
With CONNECT, there isn't anything. That is, yourGIT_PROXY_COMMAND handles talking to the proxy, then gives git itself a raw data pipe. My proxy allows CONNECT to 9418, and that's how I use it today.
If you tried to make POST work (It'd be POST, not GET, as you
need to connect up the sending side), either apache would have to front it for us, or "git-daemon --http" would have to accept the HTTP headers on before the input, and output a proper HTTP response before sending output. Seeing the headers would allow for us to vhost, even.
Hmm, but the proxy may not allow two-way communication. Does
the git protocol have more than one round-trip? That is:
Client:
POST http://server.git.host:80/projects/thisproject HTTP/1.1
Host: server.git.host
fetch-pack <sha1>
EOF
Server:
200 OK HTTP/1.1
<data>
EOFshould work, I'd think.
Joel
Signature · 9 lines
"Ninety feet between bases is perhaps as close as man has ever come to perfection." - Red Smith Joel Becker Principal Software Developer Oracle E-mail: [email protected] Phone: (650) 506-8127
Quoted 10 lines · Joel Becker
Quoted 4 lines · Linus Torvalds
If it's _just_ the initial GET/CONNECT strings, yeah, we could probably easily make the git-daemon just ignore them. That shouldn't be a problem.
But if there's anything *else* required, it gets uglier much more quickly.
With CONNECT, there isn't anything. That is, your
GIT_PROXY_COMMANDhandles talking to the proxy, then gives git itself a raw data pipe. My proxy allows CONNECT to 9418, and that's how I use it today.
Yes. Connect is easy. However many companies only allow CONNECT to 443 (not that it's much more secure than allowing it anywhere, but at least it has to block CONNECT to 25 to block sending spam).
Quoted 15 lines
If you tried to make POST work (It'd be POST, not GET, as you
need to connect up the sending side), either apache would have to front it for us, or "git-daemon --http" would have to accept the HTTP headers on before the input, and output a proper HTTP response before sending output. Seeing the headers would allow for us to vhost, even.
Hmm, but the proxy may not allow two-way communication. Does
the git protocol have more than one round-trip? That is:Client:
POST http://server.git.host:80/projects/thisproject HTTP/1.1
Host: server.git.hostfetch-pack <sha1>
EOF
Quoted 7 lines
Server:
200 OK HTTP/1.1 <data> EOFshould work, I'd think.
Well, that does not require git at all -- apache can handle this all right. But it's not network-efficient. To be network-efficient, it is necessary to negotiate the list of objects that need to be send. And that requires more than one round-trip. Additionally, the current git protocol is streaming -- the client sends data without waiting for the server. So it would require slightly different protocol over HTTP.
Signature · 1 line
Jan 'Bulb' Hudec <[email protected]>
Quoted 3 lines · Joel Becker
A normal company needs to have their firewall allow CONNECT to
9418. Then git proxying over HTTP is possible to a standard git-daemon.
443 should work too (that's HTTPS, and the proxy can't filter it, since this would be a man-in-the-middle attack).
Signature · 1 line
Matthieu
Quoted 8 lines · Linus Torvalds
Quoted 4 lines · Matthieu Moy
Many (if not most?) of the people working in a big company, I'd say. Year, it sucks, but people having used a paranoid firewall with a not-less-paranoid and broken proxy understand what I mean.
Quoted 8 lines
Well, we could try to support the git protocol over port 80..
IOW, it's probably easier to try to get people to use
git clone
git://some.host:80/projectand just run git-daemon on port 80, than it is to try to set of magic cgi scripts etc.
Except some filtering firewalls try and strip content from data (like ActiveX controls.)
Running git on port 53 will bypass pretty much every firewall out there.
(If you want to learn how to bypass an overactive firewall, talk to a bunch of teenagers at a school with an agressive porn filter.)
Signature · 4 lines
"ANSI C says access to the padding fields of a struct is undefined.
ANSI C also says that struct assignment is a memcpy. Therefore struct
assignment in ANSI C is a violation of ANSI C..."
- Alan CoxQuoted 15 lines · Linus Torvalds
Quoted 4 lines · Matthieu Moy
Many (if not most?) of the people working in a big company, I'd say. Year, it sucks, but people having used a paranoid firewall with a not-less-paranoid and broken proxy understand what I mean.
Well, we could try to support the git protocol over port 80..
IOW, it's probably easier to try to get people to use
git clone
git://some.host:80/projectand just run git-daemon on port 80, than it is to try to set of magic cgi scripts etc.
Quoted 6 lines
Doing that with virtual hosts etc should be pretty trivial. Much more so than trying to make a git-cgi script.
And yes, I do realize that in theory you can have http-aware firewalls that expect to see the normal http sequences in the first few packets in order to pass things through, but I seriously doubt it's very common.
they are actually more common than you think, and getting even more common thanks to IE
when a person browsing a hostile website will allow that website to take over the machine the demand is created for 'malware filters' for http, to do this the firewalls need to decode the http, and in the process limit you to only doing legitimate http.
it's also the case that the companies that have firewalls paranoid enough to not let you get to the git port are highly likely to be paranoid enough to have a malware filtering http firewall.
David Lang
Quoted 8 lines · [email protected] wrote:
when a person browsing a hostile website will allow that website to take over the machine the demand is created for 'malware filters' for http, to do this the firewalls need to decode the http, and in the process limit you to only doing legitimate http.
it's also the case that the companies that have firewalls paranoid enough to not let you get to the git port are highly likely to be paranoid enough to have a malware filtering http firewall.
I'm behind such a filter, and fetch git.git via HTTP just to keep my work system current with Junio. ;-)
Of course we're really really really paranoid about our firewall, but are also so paranoid that any other web browser *except* Microsoft Internet Explorer is thought to be a security risk and is more-or-less banned from the network.
The kicker is some of our developers create public websites, where testing your local webpage with Firefox and Safari is pretty much required... but those browsers still aren't as trusted as IE and require special clearances. *shakes head*
We're pretty much limited to:
*) Running the native Git protocol SSL, where the remote system is answering to port 443. It may not need to be HTTP at all, but it probably has to smell enough like SSL to get it through the malware filter. Oh, what's that? The filter cannot actually filter the SSL data? Funny! ;-)
*) Using a single POST upload followed by response from server, formatted with minimal HTTP headers. The real problem as people have pointed out is not the HTTP headers, but it is the single exchange.
One might think you could use HTTP pipelining to try and get a bi-directional channel with the remote system, but I'm sure proxy servers are not required to reuse the same TCP connection to the remote HTTP server when the inside client piplines a new request. So any sort of hack on pipelining won't work.
If you really want a stateful exchange you have to treat HTTP as though it were IP, but with reliable (and much more expensive) packet delivery, and make the Git daemon keep track of the protocol state with the client. Yes, that means that when the client suddenly goes away and doesn't tell you he went away you also have to garbage collect your state. No nice messages from your local kernel. :-(
Signature · 1 line
Shawn.
Quoted 18 lines · Shawn O. Pearce
Quoted 8 lines · [email protected] wrote:
when a person browsing a hostile website will allow that website to take over the machine the demand is created for 'malware filters' for http, to do this the firewalls need to decode the http, and in the process limit you to only doing legitimate http.
it's also the case that the companies that have firewalls paranoid enough to not let you get to the git port are highly likely to be paranoid enough to have a malware filtering http firewall.
I'm behind such a filter, and fetch git.git via HTTP just to keep my work system current with Junio. ;-)
Of course we're really really really paranoid about our firewall, but are also so paranoid that any other web browser *except* Microsoft Internet Explorer is thought to be a security risk and is more-or-less banned from the network.
Quoted 4 lines
The kicker is some of our developers create public websites, where testing your local webpage with Firefox and Safari is pretty much required... but those browsers still aren't as trusted as IE and require special clearances. *shakes head*
this isn't paranoia, this is just bullheadedness
Quoted 7 lines
We're pretty much limited to:
*) Running the native Git protocol SSL, where the remote system is answering to port 443. It may not need to be HTTP at all, but it probably has to smell enough like SSL to get it through the malware filter. Oh, what's that? The filter cannot actually filter the SSL data? Funny! ;-)
we're actually paranoid enough to have devices that do man-in-the-middle decryption for some sites, and are given copies of the encryption keys that other sites (and browsers) use so that it can decrypt the SSL and check it. I admit that this is far more paranoid then almost all sites though :-)
Quoted 11 lines
*) Using a single POST upload followed by response from server, formatted with minimal HTTP headers. The real problem as people have pointed out is not the HTTP headers, but it is the single exchange.
If you really want a stateful exchange you have to treat HTTP as though it were IP, but with reliable (and much more expensive) packet delivery, and make the Git daemon keep track of the protocol state with the client. Yes, that means that when the client suddenly goes away and doesn't tell you he went away you also have to garbage collect your state. No nice messages from your local kernel. :-(
unfortunantly you are right about this.
David Lang
Quoted 22 lines · Jan Hudec
Hello,
Did anyone already think about fetching over HTTP working similarly to the native git protocol?
That is rather than reading the raw content of the repository, there would be a CGI script (could be integrated to gitweb), that would negotiate what the client needs and then generate and send a single pack with it.
Mercurial and bzr both have this option. It would IMO have three benefits:
- Fast access for people behind paranoid firewalls, that only let http and https (you can tunel anything through, but only to port 443) through.
- Can be run on shared machine. If you have web space on machine shared by many people, you can set up your own gitweb, but cannot/are not allowed to start your own network server for git native protocol.
- Less things to set up. If you are setting up gitweb anyway, you'd not need to set up additional thing for providing fetch access.Than a question is how to implement it. The current protocol is stateful on both sides, but the stateless nature of HTTP more or less requires the protocol to be stateless on the server.
Quoted 9 lines
I think it would be possible to use basically the same protocol as now, but make it stateless for server. That is server first sends it's heads and than client repeatedly sends all it's wants and some haves until the server acks all of them and sends the pack.
Alternatively I am thinking about using Bloom filters (somebody came with such idea on the bzr list when I still followed it). It might be useful, as over HTTP we need to send as many haves as possible in one go.
Bundles?
Client POSTs it's ref set; server uses the ref set to generate and return the bundle.
Push over http(s) could work the same...
Quoted 2 lines · Jan Hudec
Did anyone already think about fetching over HTTP working similarly to the native git protocol?
Do the indexes have enough info to use them with http ranges? It'd be chunkier than a smart protocol, but it'd still work with dumb servers.
cheers,
m
Hi,
Quoted 6 lines · Martin Langhoff
Quoted 2 lines · Jan Hudec
Did anyone already think about fetching over HTTP working similarly to the native git protocol?
Do the indexes have enough info to use them with http ranges? It'd be chunkier than a smart protocol, but it'd still work with dumb servers.
It would not be really performant, would it? Besides, not all Web servers speak HTTP/1.1...
Ciao,
Dscho
Quoted 5 lines · Johannes Schindelin
Quoted 2 lines · Martin Langhoff
Do the indexes have enough info to use them with http ranges? It'd be chunkier than a smart protocol, but it'd still work with dumb servers.
It would not be really performant, would it? Besides, not all Web servers speak HTTP/1.1...
Performant compared to downloading a huge packfile to get 10% of it? Sure! It'd probably take a few trips, and you'd end up fetching 20% of the file, still better than 100%.
Quoted 1 line
Besides, not all Web servers speak HTTP/1.1...
Are there any interesting webservers out there that don't? Hand-rolled purpose-built webservers often don't but those don't serve files, they serve web apps. When it comes to serving files, any webserver that is supported (security-wise) these days is HTTP/1.1.
And for services like SF.net it'd be a safe low-cpu way of serving git files. 'cause the git protocol is quite expensive server-side (io+cpu) as we've seen with kernel.org. Being really smart with a cgi is probably going to be expensive too.
cheers,
m
Hi,
Quoted 10 lines · Martin Langhoff
Quoted 5 lines · Johannes Schindelin
Quoted 2 lines · Martin Langhoff
Do the indexes have enough info to use them with http ranges? It'd be chunkier than a smart protocol, but it'd still work with dumb servers.
It would not be really performant, would it? Besides, not all Web servers speak HTTP/1.1...
Performant compared to downloading a huge packfile to get 10% of it? Sure! It'd probably take a few trips, and you'd end up fetching 20% of the file, still better than 100%.
Don't forget that those 10% probably do not do you the favour to be in large chunks. Chances are that _every_ _single_ wanted object is separate from the others.
Quoted 11 lines
Quoted 1 line
Besides, not all Web servers speak HTTP/1.1...
Are there any interesting webservers out there that don't? Hand-rolled purpose-built webservers often don't but those don't serve files, they serve web apps. When it comes to serving files, any webserver that is supported (security-wise) these days is HTTP/1.1.
And for services like SF.net it'd be a safe low-cpu way of serving git files. 'cause the git protocol is quite expensive server-side (io+cpu) as we've seen with kernel.org. Being really smart with a cgi is probably going to be expensive too.
It's probably better and faster than relying on a feature which does not exactly help.
Ciao,
Dscho
Quoted 3 lines · Johannes Schindelin
Don't forget that those 10% probably do not do you the favour to be in large chunks. Chances are that _every_ _single_ wanted object is separate from the others.
That's completely possible. Assuming the objects even are packed in the first place. Its very unlikely that you would be able to fetch very large of a range from an existing packfile, you would be submitting most of your range requests for very very small sections.
Quoted 7 lines
Quoted 4 lines
And for services like SF.net it'd be a safe low-cpu way of serving git files. 'cause the git protocol is quite expensive server-side (io+cpu) as we've seen with kernel.org. Being really smart with a cgi is probably going to be expensive too.
It's probably better and faster than relying on a feature which does not exactly help.
Yes. Packing more often and pack v4 may help a lot there.
The other thing is kernel.org should really try to encourage the folks with repositories there to try and share against one master repository, so the poor OS has a better chance at holding the bulk of linux-2.6.git in buffer cache.
I'm not suggesting they share specifically against Linus' repository; maybe hpa and the other admins can host one seperately from Linus and enourage users to use that repository when on a system they maintain.
In an SF.net type case this doesn't help however. Most of SF.net is tiny projects with very few, if any, developers. Hence most of that is going to be unsharable, infrequently accessed, and uh, not needed to be stored in buffer cache. For the few projects that are hosted there that have a large developer base they could use a shared repository approach as I just suggested for kernel.org.
aka the "forks" thing in gitweb, and on repo.or.cz...
Signature · 1 line
Shawn.
Quoted 10 lines · Shawn O. Pearce
Quoted 7 lines · Johannes Schindelin
Quoted 4 lines
And for services like SF.net it'd be a safe low-cpu way of serving git files. 'cause the git protocol is quite expensive server-side (io+cpu) as we've seen with kernel.org. Being really smart with a cgi is probably going to be expensive too.
It's probably better and faster than relying on a feature which does not exactly help.
Quoted 6 lines
Yes. Packing more often and pack v4 may help a lot there.
The other thing is kernel.org should really try to encourage the folks with repositories there to try and share against one master repository, so the poor OS has a better chance at holding the bulk of linux-2.6.git in buffer cache.
do you mean more precisely share against one object store or do you really mean repository?
David Lang
Quoted 14 lines
I'm not suggesting they share specifically against Linus' repository; maybe hpa and the other admins can host one seperately from Linus and enourage users to use that repository when on a system they maintain.
In an SF.net type case this doesn't help however. Most of SF.net is tiny projects with very few, if any, developers. Hence most of that is going to be unsharable, infrequently accessed, and uh, not needed to be stored in buffer cache. For the few projects that are hosted there that have a large developer base they could use a shared repository approach as I just suggested for kernel.org.
aka the "forks" thing in gitweb, and on repo.or.cz...
Quoted 9 lines · [email protected] wrote:
Quoted 5 lines · Shawn O. Pearce
The other thing is kernel.org should really try to encourage the folks with repositories there to try and share against one master repository, so the poor OS has a better chance at holding the bulk of linux-2.6.git in buffer cache.
do you mean more precisely share against one object store or do you really mean repository?
Sorry, I did mean "object store". ;-)
Repository is insanity, as the refs and tags namespaces are suddenly shared. What a nightmare that would become.
Signature · 1 line
Shawn.
Quoted 8 lines · Shawn O. Pearce
Quoted 7 lines · [email protected] wrote:
Quoted 5 lines · Shawn O. Pearce
The other thing is kernel.org should really try to encourage the folks with repositories there to try and share against one master repository, so the poor OS has a better chance at holding the bulk of linux-2.6.git in buffer cache.
Quoted 4 lines
Quoted 2 lines
do you mean more precisely share against one object store or do you really mean repository?
Sorry, I did mean "object store". ;-)
And even there, I don't mean symlink objects to a shared database, I mean use the objects/info/alternates file to point to the shared, read-only location.
Its not perfect. The hotter parts of the object database is almost always the recent stuff, as that's what people are actively trying to fetch, or are using as a base when they are trying to fetch from someone else. The hotter parts are also probably too new to be in the shared store offered by kernel.org admins, which means you cannot get good IO buffering. Back to the current set of problems.
A single shared object directory that everyone can write new files into, but cannot modify or delete from, would help that problem quite a bit. But it opens up huge problems about pruning, as there is no way to perform garbage collection on that database without scanning every ref on the system, and that's just not simply possible on a busy system like kernel.org.
Signature · 1 line
Shawn.
Quoted 6 lines · Shawn O. Pearce
Its not perfect. The hotter parts of the object database is almost always the recent stuff, as that's what people are actively trying to fetch, or are using as a base when they are trying to fetch from someone else. The hotter parts are also probably too new to be in the shared store offered by kernel.org admins, which means you cannot get good IO buffering. Back to the current set of problems.
Actually, as long as objects/info/alternates is pointing at Linus's kernel.org tree, I would think that it should work relatively well, since everyone is normally basing their work on top of his tree as a starting point.
- Ted
Quoted 9 lines · Shawn O. Pearce
Quoted 3 lines · Johannes Schindelin
Don't forget that those 10% probably do not do you the favour to be in large chunks. Chances are that _every_ _single_ wanted object is separate from the others.
That's completely possible. Assuming the objects even are packed in the first place. Its very unlikely that you would be able to fetch very large of a range from an existing packfile, you would be submitting most of your range requests for very very small sections.
Well, in the commit objects case you're likely to have a bunch of them all contigous.
For tree and blob objects it is less likely.
And of course there is the question of deltas for which you might or might not have the base object locally already.
Still... I wonder if this could be actually workable. A typical daily update on the Linux kernel repository might consist of a couple hundreds or a few tousands objects. This could still be faster to fetch parts of a pack than the whole pack if the size difference is above a certain treshold. It is certainly not worse than fetching loose objects.
Things would be pretty horrid if you think of fetching a commit object, parsing it to find out what tree object to fetch, then parse that tree object to find out what other objects to fetch, and so on.
But if you only take the approach of fetching the pack index files, finding out about the objects that the remote has that are not available locally, and then fetching all those objects from within pack files without even looking at them (except for deltas), then it should be possible to issue a couple requests in parallel and possibly have decent performances. And if it turns out that more than, say, 70% of a particular pack is to be fetched (you can determine that up front), then it might be decided to fetch the whole pack.
There is no way to sensibly keep those objects packed on the receiving end of course, but storing them as loose objects and repacking them afterwards should be just fine.
Of course you'll get objects from branches in the remote repository you might not be interested in, but that's a price to pay for such a hack. On average the overhead shouldn't be that big anyway if branches within a repository are somewhat related.
I think this is something worth experimenting.
Nicolas
Hi,
Quoted 23 lines · Nicolas Pitre
Still... I wonder if this could be actually workable. A typical daily update on the Linux kernel repository might consist of a couple hundreds or a few tousands objects. This could still be faster to fetch parts of a pack than the whole pack if the size difference is above a certain treshold. It is certainly not worse than fetching loose objects.
Things would be pretty horrid if you think of fetching a commit object, parsing it to find out what tree object to fetch, then parse that tree object to find out what other objects to fetch, and so on.
But if you only take the approach of fetching the pack index files, finding out about the objects that the remote has that are not available locally, and then fetching all those objects from within pack files without even looking at them (except for deltas), then it should be possible to issue a couple requests in parallel and possibly have decent performances. And if it turns out that more than, say, 70% of a particular pack is to be fetched (you can determine that up front), then it might be decided to fetch the whole pack.
There is no way to sensibly keep those objects packed on the receiving end of course, but storing them as loose objects and repacking them afterwards should be just fine.
Quoted 6 lines
Of course you'll get objects from branches in the remote repository you might not be interested in, but that's a price to pay for such a hack. On average the overhead shouldn't be that big anyway if branches within a repository are somewhat related.
I think this is something worth experimenting.
I am a bit wary about that, because it is so complex. IMHO a cgi which gets, say, up to a hundred refs (maybe something like ref~0, ref~1, ref~2, ref~4, ref~8, ref~16, ... for the refs), and then makes a bundle for that case on the fly, is easier to do.
Of course, as with all cgi scripts, you have to make sure that DOS attacks have a low probability of success.
Ciao,
Dscho
Quoted 32 lines · Johannes Schindelin
Hi,
Quoted 28 lines · Nicolas Pitre
Still... I wonder if this could be actually workable. A typical daily update on the Linux kernel repository might consist of a couple hundreds or a few tousands objects. This could still be faster to fetch parts of a pack than the whole pack if the size difference is above a certain treshold. It is certainly not worse than fetching loose objects.
Things would be pretty horrid if you think of fetching a commit object, parsing it to find out what tree object to fetch, then parse that tree object to find out what other objects to fetch, and so on.
But if you only take the approach of fetching the pack index files, finding out about the objects that the remote has that are not available locally, and then fetching all those objects from within pack files without even looking at them (except for deltas), then it should be possible to issue a couple requests in parallel and possibly have decent performances. And if it turns out that more than, say, 70% of a particular pack is to be fetched (you can determine that up front), then it might be decided to fetch the whole pack.
There is no way to sensibly keep those objects packed on the receiving end of course, but storing them as loose objects and repacking them afterwards should be just fine.
Of course you'll get objects from branches in the remote repository you might not be interested in, but that's a price to pay for such a hack. On average the overhead shouldn't be that big anyway if branches within a repository are somewhat related.
Quoted 6 lines
Quoted 1 line
I think this is something worth experimenting.
I am a bit wary about that, because it is so complex. IMHO a cgi which gets, say, up to a hundred refs (maybe something like ref~0, ref~1, ref~2, ref~4, ref~8, ref~16, ... for the refs), and then makes a bundle for that case on the fly, is easier to do.
And if you have 1) the permission and 2) the CPU power to execute such a cgi on the server and obviously 3) the knowledge to set it up properly, then why aren't you running the Git daemon in the first place? After all, they both boil down to running git-pack-objects and sending out the result. I don't think such a solution really buys much.
On the other hand, if the client does all the work and provides the server with a list of ranges within a pack it wants to be sent, then you simply have zero special setup to perform on the hosting server and you keep the server load down due to not running pack-objects there. That, at least, is different enough from the Git daemon to be worth considering. Not only does it provide an advantage to those who cannot do anything but http out of their segregated network, but it also provide many advantages on the server side too while the cgi approach doesn't.
And actually finding out the list of objects the remote has that you don't have is not that complex. It could go as follows:
1) Fetch every .idx files the remote has.
2) From those .idx files, keep only a list of objects that are unknown locally. A good starting point for doing this really efficiently is the code for git-pack-redundant.
3) From the .idx files we got in (1), create a reverse index to get each object's size in the remote pack. The code to do this already exists in builtin-pack-objects.c.
4) With the list of missing objects from (2) along with their offset and size within a given pack file, fetch those objects from the remote server. Either perform multiple requests in parallel, or as someone mentioned already, provide the server with a list of ranges you want to be sent.
5) Store the received objects as loose objects locally. If a given object is a delta, verify if its base is available locally, or if it is listed amongst those objects to be fetched from the server. If not, add it to the list. In most cases, delta base objects will be objects already listed to be fetched anyway. To greatly simplify things, the loose delta object type from 2 years ago could be revived (commit 91d7b8afc2) since a repack will get rid of them.
6 Repeat (4) and (5) until everything has been fetched.
7) Run git-pack-objects with the list of fetched objects.
Et voilà. Oh, and of course update your local refs from the remote's.
Actually there is nothing really complex in the above operations. And with this the server side remains really simple with no special setup nor extra load beyond the simple serving of file content.
Nicolas
Quoted 7 lines · Nicolas Pitre
Quoted 1 line · Johannes Schindelin
On Wed, 16 May 2007, Nicolas Pitre wrote:
And if you have 1) the permission and 2) the CPU power to execute such a cgi on the server and obviously 3) the knowledge to set it up properly, then why aren't you running the Git daemon in the first place? After all, they both boil down to running git-pack-objects and sending out the result. I don't think such a solution really buys much.
Yes, it does. I had 2 accounts where I could run CGI, but not separate server, at university while I studied and now I can get the same on friend's server. Neither of them would probably be ok for serving larger busy git repository, but something smaller accessed by several people is OK. I think this is quite common for university students.
Of course your suggestion which moves the logic to client-side is a good one, but even the cgi with logic on server side would help in some situations.
Quoted 10 lines
On the other hand, if the client does all the work and provides the server with a list of ranges within a pack it wants to be sent, then you simply have zero special setup to perform on the hosting server and you keep the server load down due to not running pack-objects there. That, at least, is different enough from the Git daemon to be worth considering. Not only does it provide an advantage to those who cannot do anything but http out of their segregated network, but it also provide many advantages on the server side too while the cgi approach doesn't.
Quoted 4 lines
And actually finding out the list of objects the remote has that you don't have is not that complex. It could go as follows:
1) Fetch every .idx files the remote has.
... for git it's 1.2 MiB. And that definitely isn't a huge source tree. Of course the local side could remember which indices it already saw during previous fetch from that location and not re-fetch them.
A slight problem is, that git-repack normally recombines everything to a single pack, so the index would have to be re-fetched again anyway.
Quoted 8 lines
2) From those .idx files, keep only a list of objects that are unknown locally. A good starting point for doing this really efficiently is the code for git-pack-redundant.
3) From the .idx files we got in (1), create a reverse index to get each object's size in the remote pack. The code to do this already exists in
builtin-pack-objects.c.
Quoted 5 lines
4) With the list of missing objects from (2) along with their offset and size within a given pack file, fetch those objects from the remote server. Either perform multiple requests in parallel, or as someone mentioned already, provide the server with a list of ranges you want to be sent.
Does the git server really have to do so much beyond that? I didn't look at the algorithm that finds what deltas should be based on, but depending on that it might (or might not) be possible to proof the client has everything to understand if the server sends the objects as it currently has them.
Quoted 9 lines
5) Store the received objects as loose objects locally. If a given object is a delta, verify if its base is available locally, or if it is listed amongst those objects to be fetched from the server. If not, add it to the list. In most cases, delta base objects will be objects already listed to be fetched anyway. To greatly simplify things, the loose delta object type from 2 years ago could be revived (commit
91d7b8afc2) since a repack will get rid of them.6 Repeat (4) and (5) until everything has been fetched.
Unless I am really seriously missing something, there is no point in repeating. For each pack you need to unpack a delta either:
- you have it => ok.
- you don't have it, but the server does =>
but than it's already in the fetch set calculated in 2.
- you don't have it and nor does server =>
the repository at server is corrupted and you can't fix it.
Quoted 7 lines
7) Run git-pack-objects with the list of fetched objects.
Et voilà. Oh, and of course update your local refs from the remote's.
Actually there is nothing really complex in the above operations. And with this the server side remains really simple with no special setup nor extra load beyond the simple serving of file content.
On the other hand the amount of data transfered is larger, than with the git server approach, because at least the indices have to be transfered in entirety. So each approach has it's own advantages.
Signature · 1 line
Jan 'Bulb' Hudec <[email protected]>
Hi,
Quoted 9 lines · Jan Hudec
Quoted 6 lines · Nicolas Pitre
And if you have 1) the permission and 2) the CPU power to execute such a cgi on the server and obviously 3) the knowledge to set it up properly, then why aren't you running the Git daemon in the first place? After all, they both boil down to running git-pack-objects and sending out the result. I don't think such a solution really buys much.
Quoted 5 lines
Yes, it does. I had 2 accounts where I could run CGI, but not separate server, at university while I studied and now I can get the same on friend's server. Neither of them would probably be ok for serving larger busy git repository, but something smaller accessed by several people is OK. I think this is quite common for university students.
1) This has nothing to do with the way the repo is served, but how much you advertise it. The load will not be lower, just because you use a CGI script.
2) you say yourself that git-daemon would have less impact on the load:
Quoted 5 lines
Quoted 5 lines
[...]
Et voilà. Oh, and of course update your local refs from the remote's.
Quoted 7 lines
Quoted 3 lines
Actually there is nothing really complex in the above operations. And with this the server side remains really simple with no special setup nor extra load beyond the simple serving of file content.
On the other hand the amount of data transfered is larger, than with the git server approach, because at least the indices have to be transfered in entirety.
Ciao,
Dscho
Quoted 19 lines · Johannes Schindelin
Hi,
Quoted 9 lines · Jan Hudec
Quoted 6 lines · Nicolas Pitre
And if you have 1) the permission and 2) the CPU power to execute such a cgi on the server and obviously 3) the knowledge to set it up properly, then why aren't you running the Git daemon in the first place? After all, they both boil down to running git-pack-objects and sending out the result. I don't think such a solution really buys much.
Quoted 5 lines
Yes, it does. I had 2 accounts where I could run CGI, but not separate server, at university while I studied and now I can get the same on friend's server. Neither of them would probably be ok for serving larger busy git repository, but something smaller accessed by several people is OK. I think this is quite common for university students.
Quoted 3 lines
1) This has nothing to do with the way the repo is served, but how much you advertise it. The load will not be lower, just because you use a CGI script.
That won't. But that was never the purpose of "smart cgi". The purpose was to minimize the bandwidth usage (and connectivity is still not so cheap that you'd not care) while still working over http either because the users need to access it from behind firewall or because administrator is not willing to set up git-daemon for you, while CGI you can run yourself.
Quoted 1 line
2) you say yourself that git-daemon would have less impact on the load:
NO, I didn't -- at least not in the paragraph below.
In the below paragraph I said, that *network* use will never be as good with *dumb* solution, as it can be with smart solution, no matter whether it is over special protocol or HTTP.
---
Of course it would be less efficient in both CPU and network load, because there is the overhead of the web server and overhead of the http headers.
Actually I like the ranges solution. If accompanied with repack stategy that does not pack everything together, but instead creates packs of limited number of objects -- so that the indices don't exceed configurable size, say 64kB -- could not so much less efficient for the network and have the advantage of working without ability to execute CGI.
Quoted 12 lines
Quoted 12 lines
Quoted 8 lines
[...]
Et voilà. Oh, and of course update your local refs from the remote's.
Actually there is nothing really complex in the above operations. And with this the server side remains really simple with no special setup nor extra load beyond the simple serving of file content.
On the other hand the amount of data transfered is larger, than with the git server approach, because at least the indices have to be transfered in entirety.
Signature · 1 line
Jan 'Bulb' Hudec <[email protected]>
Quoted 9 lines · Jan Hudec
Quoted 7 lines · Nicolas Pitre
Quoted 1 line · Johannes Schindelin
On Wed, 16 May 2007, Nicolas Pitre wrote:
And if you have 1) the permission and 2) the CPU power to execute such a cgi on the server and obviously 3) the knowledge to set it up properly, then why aren't you running the Git daemon in the first place? After all, they both boil down to running git-pack-objects and sending out the result. I don't think such a solution really buys much.
Quoted 8 lines
Yes, it does. I had 2 accounts where I could run CGI, but not separate server, at university while I studied and now I can get the same on friend's server. Neither of them would probably be ok for serving larger busy git repository, but something smaller accessed by several people is OK. I think this is quite common for university students.
Of course your suggestion which moves the logic to client-side is a good one, but even the cgi with logic on server side would help in some situations.
You could simply wrap git-bundle within a cgi. That is certainly easy enough.
Quoted 10 lines
Quoted 10 lines
On the other hand, if the client does all the work and provides the server with a list of ranges within a pack it wants to be sent, then you simply have zero special setup to perform on the hosting server and you keep the server load down due to not running pack-objects there. That, at least, is different enough from the Git daemon to be worth considering. Not only does it provide an advantage to those who cannot do anything but http out of their segregated network, but it also provide many advantages on the server side too while the cgi approach doesn't.
Quoted 8 lines
Quoted 4 lines
And actually finding out the list of objects the remote has that you don't have is not that complex. It could go as follows:
1) Fetch every .idx files the remote has.
... for git it's 1.2 MiB. And that definitely isn't a huge source tree. Of course the local side could remember which indices it already saw during previous fetch from that location and not re-fetch them.
Right. The name of the pack/index plus its time stamp can be cached. If the remote doesn't repack too often then the overhead would be minimal.
Quoted 8 lines
Quoted 8 lines
2) From those .idx files, keep only a list of objects that are unknown locally. A good starting point for doing this really efficiently is the code for git-pack-redundant.
3) From the .idx files we got in (1), create a reverse index to get each object's size in the remote pack. The code to do this already exists in
builtin-pack-objects.c.
Quoted 7 lines
Quoted 5 lines
4) With the list of missing objects from (2) along with their offset and size within a given pack file, fetch those objects from the remote server. Either perform multiple requests in parallel, or as someone mentioned already, provide the server with a list of ranges you want to be sent.
Does the git server really have to do so much beyond that?
Yes it does. The real thing perform a full object reachability walk and only the objects that are needed for the wanted branch(es) are sent in a custom pack meaning that the data transfer is really optimal.
Quoted 10 lines
Quoted 9 lines
5) Store the received objects as loose objects locally. If a given object is a delta, verify if its base is available locally, or if it is listed amongst those objects to be fetched from the server. If not, add it to the list. In most cases, delta base objects will be objects already listed to be fetched anyway. To greatly simplify things, the loose delta object type from 2 years ago could be revived (commit
91d7b8afc2) since a repack will get rid of them.6 Repeat (4) and (5) until everything has been fetched.
Quoted 7 lines
Unless I am really seriously missing something, there is no point in repeating. For each pack you need to unpack a delta either:
- you have it => ok.
- you don't have it, but the server does =>
but than it's already in the fetch set calculated in 2.
- you don't have it and nor does server =>
the repository at server is corrupted and you can't fix it.
You're right of course.
Nicolas
Quoted 17 lines · Nicolas Pitre
Quoted 15 lines · Jan Hudec
Quoted 7 lines · Nicolas Pitre
Quoted 1 line · Johannes Schindelin
On Wed, 16 May 2007, Nicolas Pitre wrote:
And if you have 1) the permission and 2) the CPU power to execute such a cgi on the server and obviously 3) the knowledge to set it up properly, then why aren't you running the Git daemon in the first place? After all, they both boil down to running git-pack-objects and sending out the result. I don't think such a solution really buys much.
Yes, it does. I had 2 accounts where I could run CGI, but not separate server, at university while I studied and now I can get the same on friend's server. Neither of them would probably be ok for serving larger busy git repository, but something smaller accessed by several people is OK. I think this is quite common for university students.
Quoted 5 lines
Quoted 2 lines
Of course your suggestion which moves the logic to client-side is a good one, but even the cgi with logic on server side would help in some situations.
You could simply wrap git-bundle within a cgi. That is certainly easy enough.
isn't this (or something very similar) exactly what we want for a smalrt fetch via http?
after all, we're completely in control of the client software, and the useual reason for HTTP-only access is on the client side rather then the server side. so http access that wraps the git protocol in http would make life much cleaner for lots of people
there are a few cases where all you have is static web space, but I don't think it's worth trying to optimize that too much as you still have the safety issues to worry about
David Lang
Quoted 3 lines · Nicolas Pitre
And if you have 1) the permission and 2) the CPU power to execute such a cgi on the server and obviously 3) the knowledge to set it up properly, then why aren't you running the Git daemon in the first place?
And you probably _are_ running git daemon. But some clients may be on shitty connections that only allow http. That's one of the scenarios we're discussing.
cheers,
m
Quoted 8 lines · Martin Langhoff
Quoted 3 lines · Nicolas Pitre
And if you have 1) the permission and 2) the CPU power to execute such a cgi on the server and obviously 3) the knowledge to set it up properly, then why aren't you running the Git daemon in the first place?
And you probably _are_ running git daemon. But some clients may be on shitty connections that only allow http. That's one of the scenarios we're discussing.
That's not what I'm disputing at all.
I'm disputing the vertue of an HTTP solution involving a cgi with Git bundles vs an HTTP solution involving static file range serving. The clients on shitty connections don't care either ways.
Nicolas
Quoted 11 lines · Johannes Schindelin
Hi,
Quoted 7 lines · Martin Langhoff
Quoted 5 lines · Johannes Schindelin
Quoted 2 lines · Martin Langhoff
Do the indexes have enough info to use them with http ranges? It'd be chunkier than a smart protocol, but it'd still work with dumb servers.
It would not be really performant, would it? Besides, not all Web servers speak HTTP/1.1...
Quoted 7 lines
Quoted 3 lines
Performant compared to downloading a huge packfile to get 10% of it? Sure! It'd probably take a few trips, and you'd end up fetching 20% of the file, still better than 100%.
Don't forget that those 10% probably do not do you the favour to be in large chunks. Chances are that _every_ _single_ wanted object is separate from the others.
FYI, bzr uses HTTP range requests, and the introduction of this feature lead to significant performance improvement for them (bzr is more dumb-protocol oriented than git is, so that's really important there). They have this "index file+data file" system too, so you download the full index file, and then send an HTTP range request to get only the relevant parts of the data file.
The thing is, AAUI, they don't send N range requests to get N chunks, but one HTTP request, requesting the N ranges at a time, and get the N chunks a a whole (IIRC, a kind of MIME-encoded response from the server). So, you pay the price of a longer HTTP request, but not the price of N networks round-trips.
That's surely not as efficient as anything smart on the server, but might really help for the cases where the server is /not/ smart.
Signature · 1 line
Matthieu
Quoted 6 lines · Matthieu Moy
FYI, bzr uses HTTP range requests, and the introduction of this feature lead to significant performance improvement for them (bzr is more dumb-protocol oriented than git is, so that's really important there). They have this "index file+data file" system too, so you download the full index file, and then send an HTTP range request to get only the relevant parts of the data file.
That's the kind of thing I was imagining. Between the index and an additional "index-supplement-for-dumb-protocols" maintained by update-server-info, http ranges can be bent to our evil purposes.
Of course it won't be as network-efficient as the git proto, or even as the git-over-cgi proto, but it'll surely be server-cpu-and-memory efficient. And people will benefit from it without having to do any additional setup.
It might be hard to come up with a usable approach to http ranges. But I do think it's worth considering carefully.
cheers,
m
Hi,
[I missed this mail, because Matthieu culled the Cc list again]
Quoted 9 lines · Martin Langhoff
Quoted 6 lines · Matthieu Moy
FYI, bzr uses HTTP range requests, and the introduction of this feature lead to significant performance improvement for them (bzr is more dumb-protocol oriented than git is, so that's really important there). They have this "index file+data file" system too, so you download the full index file, and then send an HTTP range request to get only the relevant parts of the data file.
Quoted 8 lines
That's the kind of thing I was imagining. Between the index and an additional "index-supplement-for-dumb-protocols" maintained by update-server-info, http ranges can be bent to our evil purposes.
Of course it won't be as network-efficient as the git proto, or even as the git-over-cgi proto, but it'll surely be server-cpu-and-memory efficient. And people will benefit from it without having to do any additional setup.
Of course, the problem is that only the server can know beforehand which objects are needed. Imagine this:
X - Y - Z
\
A
Client has "X", wants "Z", but not "A". Client needs "Y" and "Z". But client cannot know that it needs "Y" before getting "Z", except if the server says so.
If you have a solution for that problem, please enlighten me: I don't.
Ciao,
Dscho
Quoted 1 line · Johannes Schindelin
If you have a solution for that problem, please enlighten me: I don't.
Ok - worst case scenario - have a minimal hints file that tells me the ranges to fetch all commits and all trees. To reduce that Add to the hints file data to name the hashes (or even better - offsets) for the delta chains that contain commits+trees relevant to all the heads - minus 10, 20, 30, 40 commits and 1,2,4,8 and 16 days.
So there's a good chance the client can get the commits+trees needed efficiently. For blobs, all you need is the index to mark the delta chains you need.
cheers,
m
Quoted 8 lines · Martin Langhoff
Quoted 1 line · Johannes Schindelin
If you have a solution for that problem, please enlighten me: I don't.
Ok - worst case scenario - have a minimal hints file that tells me the ranges to fetch all commits and all trees. To reduce that Add to the hints file data to name the hashes (or even better - offsets) for the delta chains that contain commits+trees relevant to all the heads - minus 10, 20, 30, 40 commits and 1,2,4,8 and 16 days.
NO !
This is unreliable, unnecessary, and actually kills the beauty of the solution's simplicity.
You get updates for every branches the remote has, period.
No server side extra files, no guesses, no arbitrary ranges, no backward compatibility issues, no crap!
Nicolas
Quoted 9 lines · Martin Langhoff
Quoted 1 line · Johannes Schindelin
If you have a solution for that problem, please enlighten me: I don't.
Ok - worst case scenario - have a minimal hints file that tells me the ranges to fetch all commits and all trees. To reduce that Add to the hints file data to name the hashes (or even better - offsets) for the delta chains that contain commits+trees relevant to all the heads - minus 10, 20, 30, 40 commits and 1,2,4,8 and 16 days.
Quoted 3 lines
So there's a good chance the client can get the commits+trees needed efficiently. For blobs, all you need is the index to mark the delta chains you need.
By the way, I think we always should get the whole delta chain, unless we are absolutely sure that we have base object(s) in repo.
Signature · 3 lines
Jakub Narebski Warsaw, Poland ShadeHawk on #git
Quoted 19 lines · Johannes Schindelin
Hi,
[I missed this mail, because Matthieu culled the Cc list again]
Quoted 13 lines · Martin Langhoff
Quoted 6 lines · Matthieu Moy
FYI, bzr uses HTTP range requests, and the introduction of this feature lead to significant performance improvement for them (bzr is more dumb-protocol oriented than git is, so that's really important there). They have this "index file+data file" system too, so you download the full index file, and then send an HTTP range request to get only the relevant parts of the data file.
That's the kind of thing I was imagining. Between the index and an additional "index-supplement-for-dumb-protocols" maintained by update-server-info, http ranges can be bent to our evil purposes.
Quoted 7 lines
Quoted 4 lines
Of course it won't be as network-efficient as the git proto, or even as the git-over-cgi proto, but it'll surely be server-cpu-and-memory efficient. And people will benefit from it without having to do any additional setup.
Of course, the problem is that only the server can know beforehand which objects are needed.
But the whole idea is that we don't care.
Quoted 7 lines
Imagine this:
X - Y - Z
\
A
Quoted 5 lines
Client has "X", wants "Z", but not "A". Client needs "Y" and "Z". But client cannot know that it needs "Y" before getting "Z", except if the server says so.
If you have a solution for that problem, please enlighten me: I don't.
We're talking about a _dumb_ protocol here. If you want something fancy, just use the Git daemon.
Otherwise, you'll simply get everything the remote has that you don't have, including A.
In practice this shouldn't be a problem because people tend to have clean repositories on machines they want their stuff to be published, meaning that those public repos are usually the result of pushes, hence they contain only the minimum set of needed objects. Of course you get every branches and not only a particular one, but that's the price to pay with a dumb protocol.
Nicolas
Quoted 3 lines · Johannes Schindelin
Hi,
[I missed this mail, because Matthieu culled the Cc list again]
Sorry about that, miss-configuration of my mailer. I didn't find time to solve it before.
OTOH, since most people actually complain when you Cc them on a mailing list, the choice "To Cc or not to Cc" has no universal solution ;-).
Signature · 1 line
Matthieu
Quoted 10 lines · Martin Langhoff
Quoted 5 lines · Johannes Schindelin
Quoted 2 lines · Martin Langhoff
Do the indexes have enough info to use them with http ranges? It'd be chunkier than a smart protocol, but it'd still work with dumb servers.
It would not be really performant, would it? Besides, not all Web servers speak HTTP/1.1...
Performant compared to downloading a huge packfile to get 10% of it? Sure! It'd probably take a few trips, and you'd end up fetching 20% of the file, still better than 100%.
That's why you should have something akin to backup policy for pack files, like daily packs, weekly packs, ..., and the rest, just for the dumb protocols.
Signature · 3 lines
Jakub Narebski Warsaw, Poland ShadeHawk on #git