Skip to contentSkip to conversation

problem cloning via http since v1.6.6-rc0

PATCH In review Yaroslav Halchenko 8 participants 28 messages Jun 15, 2016

lore
Share & tools
lore
#0 lore raw

Dear Git Developers,

Some users of our project started recently to complain that they could not clone the repository via http (git:// wasn't a choice due to heavy firewalling) and because http:// was used as a protocol to get sources in some distributions (e.g. macports).

Cloning of the repository works fine with v1.6.5.7 but fails with v1.6.6-rc0. I haven't done full bisection since that repository is relatively bulky and poor server is quite loaded anyways, so I thought you just would get a clue without going brute-force. But here are the details: in case of failing operation, I immediately get failure:

$> GIT_TRACE=2 ./git clone http://git.debian.org/git/pkg-exppsy/pymvpa.git
trace: built-in: git 'clone' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git'
warning: templates not found /home/yoh/share/git-core/templates
Initialized empty Git repository in /home/yoh/proj/misc/git/pymvpa/.git/
trace: run_command: 'remote-curl' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git'
trace: exec: 'git' 'remote-curl' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git'
trace: exec: 'git-remote-curl' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git'
trace: run_command: 'git-remote-curl' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git'
fatal: http://git.debian.org/git/pkg-exppsy/pymvpa.git/info/refs?service=git-upload-pack not found: did you run git update-server-info on the server?

on the server, 1.6.3.3 version of git was used to run git update-server-info.

Thanks in advance

Signature · 5 lines
Yaroslav O. Halchenko
Postdoctoral Fellow,   Department of Psychological and Brain Sciences
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik        
#1 lore raw

On Wed, Jan 20, 2010 at 07:47:56PM -0500, Yaroslav Halchenko wrote:

Added spearce to cc.

Quoted 13 lines

Dear Git Developers,

Some users of our project started recently to complain that they could not clone the repository via http (git:// wasn't a choice due to heavy firewalling) and because http:// was used as a protocol to get sources in some distributions (e.g. macports).

Cloning of the repository works fine with v1.6.5.7 but fails with v1.6.6-rc0. I haven't done full bisection since that repository is relatively bulky and poor server is quite loaded anyways, so I thought you just would get a clue without going brute-force. But here are the details: in case of failing operation, I immediately get failure:

Quoted 9 lines

$> GIT_TRACE=2 ./git clone http://git.debian.org/git/pkg-exppsy/pymvpa.git
trace: built-in: git 'clone' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git'
warning: templates not found /home/yoh/share/git-core/templates
Initialized empty Git repository in /home/yoh/proj/misc/git/pymvpa/.git/
trace: run_command: 'remote-curl' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git'
trace: exec: 'git' 'remote-curl' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git'
trace: exec: 'git-remote-curl' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git'
trace: run_command: 'git-remote-curl' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git'
fatal: http://git.debian.org/git/pkg-exppsy/pymvpa.git/info/refs?service=git-upload-pack not found: did you run git update-server-info on the server?

Looks like remote-curl (which handles http) issues request for:

'.../info/refs?service=git-upload-pack'

And expects that if there is no smart HTTP server there for the request to be interpretted as:

'.../info/refs'

(i.e. webserver would ignore the query). This isn't true for git.debian.org. Requesting the latter works (and the data formatting looks sane), but the former is 404. This causes the fetch to fail.

-Ilari

#2 lore raw

Hi,

Quoted 4 lines · Ilari Liusvaara

Looks like remote-curl (which handles http) issues request for:

'.../info/refs?service=git-upload-pack'

Quoted 8 lines

And expects that if there is no smart HTTP server there for the request to be interpretted as:

'.../info/refs'

(i.e. webserver would ignore the query). This isn't true for git.debian.org. Requesting the latter works (and the data formatting looks sane), but the former is 404. This causes the fetch to fail.

afaik, putting a "?var1=val1&var2=...." still makes it a normal GET request, even if the url requested is just a plain file and not some cgi handler that uses those variables/values.

--
Cheers,
Ray Chuan

#3 lore raw
Quoted 11 lines · Tay Ray Chuan

Hi,

On Thu, Jan 21, 2010 at 1:08 PM, Ilari Liusvaara

Quoted 4 lines

(i.e. webserver would ignore the query). This isn't true for git.debian.org. Requesting the latter works (and the data formatting looks sane), but the former is 404. This causes the fetch to fail.

afaik, putting a "?var1=val1&var2=...." still makes it a normal GET request, even if the url requested is just a plain file and not some cgi handler that uses those variables/values.

Yes, it is normal GET (POST would be something else). And wheither it is CGI doesn't come into play for request since client decides wheither to send GET or POST and wheither to include query or not.

Query is just technical name for part between ? and # (or end of HTTP URL), and can be present in any type of request that accepts http:// URL.

As said, code expects query part to be ignored if target is regular file but broke when it didn't get ignored.

-Ilari

#4 lore raw

Hi,

Quoted 13 lines · Ilari Liusvaara
Quoted 11 lines · Tay Ray Chuan

Hi,

On Thu, Jan 21, 2010 at 1:08 PM, Ilari Liusvaara

Quoted 4 lines

(i.e. webserver would ignore the query). This isn't true for git.debian.org. Requesting the latter works (and the data formatting looks sane), but the former is 404. This causes the fetch to fail.

afaik, putting a "?var1=val1&var2=...." still makes it a normal GET request, even if the url requested is just a plain file and not some cgi handler that uses those variables/values.

Quoted 6 lines

Yes, it is normal GET (POST would be something else). And wheither it is CGI doesn't come into play for request since client decides wheither to send GET or POST and wheither to include query or not.

Query is just technical name for part between ? and # (or end of HTTP URL), and can be present in any type of request that accepts http:// URL.

yes, indeed, I misread your message. Your idea of the query string affecting the server response didn't strike me then.

Signature · 2 lines
Cheers,
Ray Chuan
#5 lore raw
In reply to Tay Ray Chuan

Hi,

Quoted 11 lines · On Thu, 21 Jan 2010 14:47:37 +0800 Tay Ray Chuan <[email protected]> wrote:
Quoted 9 lines · Ilari Liusvaara

Looks like remote-curl (which handles http) issues request for:

'.../info/refs?service=git-upload-pack'

And expects that if there is no smart HTTP server there for the request to be interpretted as:

'.../info/refs'

Quoted 7 lines
Quoted 3 lines

(i.e. webserver would ignore the query). This isn't true for git.debian.org. Requesting the latter works (and the data formatting looks sane), but the former is 404. This causes the fetch to fail.

afaik, putting a "?var1=val1&var2=...." still makes it a normal GET request, even if the url requested is just a plain file and not some cgi handler that uses those variables/values.

Yaroslav, sorry for making you run in circles - it really is git's fault (sorta).

In recent versions of git, we were sending out the GET request for info/refs with a query string (?serivce=<service name>). I'm not sure why, but your server is not playing nice when the query string is appended.

Could you try this patch and see if it solves the issue? I manage to clone your repo successfully with it.

Signature · 18 lines
Cheers,
Ray Chuan

-->8--
Subject: [PATCH] http/remote-curl: coddle picky servers

When "info/refs" is a static file and not behind a CGI handler, some
servers may not handle a GET request for it with a query string
appended (eg. "?foo=bar") properly.

If such a request fails, retry it sans the query string, and also
discount the possibility of using the "smart" protocol (since no
service is specified with "?service=<service name>").

Signed-off-by: Tay Ray Chuan <[email protected]>
---
 remote-curl.c |   18 ++++++++++++++++--
 1 files changed, 16 insertions(+), 2 deletions(-)
remote-curl.c +16 -2
index 1361006..a904164 100644
@@ -102,7 +102,7 @@ static struct discovery* discover_refs(const char *service)
 	struct strbuf buffer = STRBUF_INIT;
 	struct discovery *last = last_discovery;
 	char *refs_url;
-	int http_ret, is_http = 0;
+	int http_ret, is_http = 0, proto_git_candidate = 1;
 
 	if (last && !strcmp(service, last->service))
 		return last;
@@ -121,6 +121,19 @@ static struct discovery* discover_refs(const char *service)
 
 	init_walker();
 	http_ret = http_get_strbuf(refs_url, &buffer, HTTP_NO_CACHE);
+
+	/* try again with "plain" url (no ? or & appended) */
+	if (http_ret != HTTP_OK) {
+		free(refs_url);
+		strbuf_reset(&buffer);
+
+		proto_git_candidate = 0;
+		strbuf_addf(&buffer, "%s/info/refs", url);
+		refs_url = strbuf_detach(&buffer, NULL);
+
+		http_ret = http_get_strbuf(refs_url, &buffer, HTTP_NO_CACHE);
+	}
+
 	switch (http_ret) {
 	case HTTP_OK:
 		break;
@@ -137,7 +150,8 @@ static struct discovery* discover_refs(const char *service)
 	last->buf_alloc = strbuf_detach(&buffer, &last->len);
 	last->buf = last->buf_alloc;
 
-	if (is_http && 5 <= last->len && last->buf[4] == '#') {
+	if (is_http && proto_git_candidate
+		&& 5 <= last->len && last->buf[4] == '#') {
 		/* smart HTTP response; validate that the service
 		 * pkt-line matches our request.
 		 */
Signature · 1 line
1.6.6.1.337.g96bc8
#6 lore raw

Hi Tay Ray,

That patch works fine for me ;) I only hope it would get accepted into bugfix and next dev release (I guess it might annoy some of apache admins a bit due to increase of their errors.log now even for well maintained repositories, but well -- that is life ;-) )

Thanks!
Yarik

Quoted 14 lines · Tay Ray Chuan
Quoted 3 lines

afaik, putting a "?var1=val1&var2=...." still makes it a normal GET request, even if the url requested is just a plain file and not some cgi handler that uses those variables/values.

Yaroslav, sorry for making you run in circles - it really is git's fault (sorta).

In recent versions of git, we were sending out the GET request for info/refs with a query string (?serivce=<service name>). I'm not sure why, but your server is not playing nice when the query string is appended.

Could you try this patch and see if it solves the issue? I manage to clone your repo successfully with it.

Signature · 5 lines
                                  .-.
=------------------------------   /v\  ----------------------------=
Keep in touch                    // \\     (yoh@|www.)onerussian.com
Yaroslav Halchenko              /(   )\               ICQ#: 60653192
                   Linux User    ^^-^^    [175555]
#7 lore raw

PATCH 1/1 http/remote-curl: coddle picky servers

When "info/refs" is a static file and not behind a CGI handler, some servers may not handle a GET request for it with a query string appended (eg. "?foo=bar") properly.

If such a request fails, retry it sans the query string. In addition, ensure that the "smart" http protocol is not used (a service has to be specified with "?service=<service name>" to be conformant).

---
 remote-curl.c |   18 ++++++++++++++++--
 1 files changed, 16 insertions(+), 2 deletions(-)
remote-curl.c +16 -2
index 1361006..a904164 100644
@@ -102,7 +102,7 @@ static struct discovery* discover_refs(const char *service)
 	struct strbuf buffer = STRBUF_INIT;
 	struct discovery *last = last_discovery;
 	char *refs_url;
-	int http_ret, is_http = 0;
+	int http_ret, is_http = 0, proto_git_candidate = 1;
 
 	if (last && !strcmp(service, last->service))
 		return last;
@@ -121,6 +121,19 @@ static struct discovery* discover_refs(const char *service)
 
 	init_walker();
 	http_ret = http_get_strbuf(refs_url, &buffer, HTTP_NO_CACHE);
+
+	/* try again with "plain" url (no ? or & appended) */
+	if (http_ret != HTTP_OK) {
+		free(refs_url);
+		strbuf_reset(&buffer);
+
+		proto_git_candidate = 0;
+		strbuf_addf(&buffer, "%s/info/refs", url);
+		refs_url = strbuf_detach(&buffer, NULL);
+
+		http_ret = http_get_strbuf(refs_url, &buffer, HTTP_NO_CACHE);
+	}
+
 	switch (http_ret) {
 	case HTTP_OK:
 		break;
@@ -137,7 +150,8 @@ static struct discovery* discover_refs(const char *service)
 	last->buf_alloc = strbuf_detach(&buffer, &last->len);
 	last->buf = last->buf_alloc;
 
-	if (is_http && 5 <= last->len && last->buf[4] == '#') {
+	if (is_http && proto_git_candidate
+		&& 5 <= last->len && last->buf[4] == '#') {
 		/* smart HTTP response; validate that the service
 		 * pkt-line matches our request.
 		 */

--
1.6.6.1.337.g96bc8

Acked-by on v1 1/1
#8 lore raw

Re: [PATCH] http/remote-curl: coddle picky servers

Quoted 10 lines · Tay Ray Chuan

When "info/refs" is a static file and not behind a CGI handler, some servers may not handle a GET request for it with a query string appended (eg. "?foo=bar") properly.

If such a request fails, retry it sans the query string. In addition, ensure that the "smart" http protocol is not used (a service has to be specified with "?service=<service name>" to be conformant).

*grumble* stupid Apache *grumble*

Signature · 1 line
Shawn.
#9 lore raw

Re: [PATCH] http/remote-curl: coddle picky servers

Quoted 5 lines · Shawn O. Pearce
Quoted 4 lines · Tay Ray Chuan

When "info/refs" is a static file and not behind a CGI handler, some servers may not handle a GET request for it with a query string appended (eg. "?foo=bar") properly.

Quoted 8 lines
Quoted 6 lines

If such a request fails, retry it sans the query string. In addition, ensure that the "smart" http protocol is not used (a service has to be specified with "?service=<service name>" to be conformant).

*grumble* stupid Apache *grumble*

stupid Apache... configuration.

Check the error message you get on
http://git.debian.org/git/pkg-exppsy/pymvpa.git/info/refs?service=git-upload-pack:

The requested URL /gitweb.cgigit/pkg-exppsy/pymvpa.git/info/refs was not found on this server.

Look closely at the start of the requested URL: /gitweb.cgi... It comes from this rule:

RewriteCond %{QUERY_STRING} ^(.+)$
RewriteRule ^/(.*)$ /gitweb.cgi$1 [L,PT]

which is global to the virtual host.

Anyways, while git.debian.org can certainly be fixed for that, other servers may want to do some different things with urls with parameters.

Mike

#10 lore raw

git fetch -v not at all verbose?

Hi!
On many of my trees (with linux kernel), git fetch is slower than git clone. Even more annoyingly, it would hang sometimes for tens of minutes without any output, even if -v is supplied.

stracing it shows a ton of lines like the following:
16324 read(10, "ACK 4bbdfe65d23014f539fec4227260"..., 51) = 51
16324 read(10, "0037", 4) = 4
16324 read(10, "ACK 322c06560fa314b04a6302ea03c0"..., 51) = 51
16324 read(10, "0037", 4) = 4
16324 read(10, "ACK 848ea2043b128b5947851866a114"..., 51) = 51
16324 read(10, "0037", 4) = 4

Is there some way to make got fetch show progress at this stage, or even better, can it be made faster somehow?

Thanks!

Signature · 1 line
MST
#11 lore raw

Re: git fetch -v not at all verbose?

Quoted 3 lines · Michael S. Tsirkin

On many of my trees (with linux kernel), git fetch is slower than git clone. Even more annoyingly, it would hang sometimes for tens of minutes without any output, even if -v is supplied.

Ouch. I think -v -v boosts the output to be ever more verbose, and might actually show you something.

Quoted 7 lines

stracing it shows a ton of lines like the following:
16324 read(10, "ACK 4bbdfe65d23014f539fec4227260"..., 51) = 51
16324 read(10, "0037", 4) = 4
16324 read(10, "ACK 322c06560fa314b04a6302ea03c0"..., 51) = 51
16324 read(10, "0037", 4) = 4
16324 read(10, "ACK 848ea2043b128b5947851866a114"..., 51) = 51
16324 read(10, "0037", 4) = 4

That's the peers trying to determine a common base.

Quoted 2 lines

Is there some way to make got fetch show progress at this stage, or even better, can it be made faster somehow?

We shouldn't need to show progress here, we should just be faster.

Given the symptom, it sounds to me like your local repository is some 1,000s of commits ahead of the remote repository you are fetching from. Is that true?

Are you fetching from a configured remote that has tracking branches, or are you fetching through a one-shot URL pasted onto the command line?

Signature · 1 line
Shawn.
#12 lore raw

Re: git fetch -v not at all verbose?

Quoted 18 lines · Shawn O. Pearce
Quoted 3 lines · Michael S. Tsirkin

On many of my trees (with linux kernel), git fetch is slower than git clone. Even more annoyingly, it would hang sometimes for tens of minutes without any output, even if -v is supplied.

Ouch. I think -v -v boosts the output to be ever more verbose, and might actually show you something.

Quoted 7 lines

stracing it shows a ton of lines like the following:
16324 read(10, "ACK 4bbdfe65d23014f539fec4227260"..., 51) = 51
16324 read(10, "0037", 4) = 4
16324 read(10, "ACK 322c06560fa314b04a6302ea03c0"..., 51) = 51
16324 read(10, "0037", 4) = 4
16324 read(10, "ACK 848ea2043b128b5947851866a114"..., 51) = 51
16324 read(10, "0037", 4) = 4

That's the peers trying to determine a common base.

Quoted 8 lines
Quoted 2 lines

Is there some way to make got fetch show progress at this stage, or even better, can it be made faster somehow?

We shouldn't need to show progress here, we should just be faster.

Given the symptom, it sounds to me like your local repository is some 1,000s of commits ahead of the remote repository you are fetching from. Is that true?

Hmm, no, but what is true is that I fetched several remotes that diverged significantly into the same local repository. Would that have same effect?

Quoted 3 lines

Are you fetching from a configured remote that has tracking branches, or are you fetching through a one-shot URL pasted onto the command line?

Configured remote.

Quoted 2 lines
Signature · 1 line
Shawn.
#13 lore raw

Re: git fetch -v not at all verbose?

Quoted 5 lines · Michael S. Tsirkin
Quoted 4 lines · Shawn O. Pearce
Quoted 3 lines · Michael S. Tsirkin

On many of my trees (with linux kernel), git fetch is slower than git clone. Even more annoyingly, it would hang sometimes for tens of minutes without any output, even if -v is supplied.

...

Quoted 7 lines
Quoted 3 lines

Given the symptom, it sounds to me like your local repository is some 1,000s of commits ahead of the remote repository you are fetching from. Is that true?

Hmm, no, but what is true is that I fetched several remotes that diverged significantly into the same local repository. Would that have same effect?

Yes.

Quoted 5 lines
Quoted 3 lines

Are you fetching from a configured remote that has tracking branches, or are you fetching through a one-shot URL pasted onto the command line?

Configured remote.

Hmm. I wonder if we should try to shortcut the commit walking in a case like this and just feed the tracking branches we already have.

Signature · 1 line
Shawn.
#14 lore raw

Re: git fetch -v not at all verbose?

Quoted 17 lines · Shawn O. Pearce
Quoted 5 lines · Michael S. Tsirkin
Quoted 4 lines · Shawn O. Pearce
Quoted 3 lines · Michael S. Tsirkin

On many of my trees (with linux kernel), git fetch is slower than git clone. Even more annoyingly, it would hang sometimes for tens of minutes without any output, even if -v is supplied.

...

Quoted 7 lines
Quoted 3 lines

Given the symptom, it sounds to me like your local repository is some 1,000s of commits ahead of the remote repository you are fetching from. Is that true?

Hmm, no, but what is true is that I fetched several remotes that diverged significantly into the same local repository. Would that have same effect?

Yes.

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

Are you fetching from a configured remote that has tracking branches, or are you fetching through a one-shot URL pasted onto the command line?

Configured remote.

Hmm. I wonder if we should try to shortcut the commit walking in a case like this and just feed the tracking branches we already have.

Or for the case of 1,000s of commits ahead, git could try to implement a heuristic to reduce the number of commits sent. Currently all commits are sent in order, correct? How about binary search like what git bisect does?

Quoted 2 lines
Signature · 1 line
Shawn.
#15 lore raw

Re: git fetch -v not at all verbose?

Quoted 5 lines · On Thursday 21 January 2010 18:30:10 Michael S. Tsirkin wrote:
Quoted 4 lines · Shawn O. Pearce
Quoted 3 lines · Michael S. Tsirkin

Hmm, no, but what is true is that I fetched several remotes that diverged significantly into the same local repository. Would that have same effect?

[...]

Quoted 7 lines
Quoted 2 lines

Hmm. I wonder if we should try to shortcut the commit walking in a case like this and just feed the tracking branches we already have.

Or for the case of 1,000s of commits ahead, git could try to implement a heuristic to reduce the number of commits sent. Currently all commits are sent in order, correct? How about binary search like what git bisect does?

I had a patch for this ages ago (that combines exponential-stride backwards search and later bisection), but it was shot down on grounds of not working at times and code convolution and I forgot about it...

I can give this another shot, but it seems most of the code has moved due to the transport handlers changes, so I'll first have to read into it again.

Signature · 2 lines
Thomas Rast
trast@{inf,student}.ethz.ch
#16 lore raw

Re: git fetch -v not at all verbose?

In reply to Shawn O. Pearce
Quoted 8 lines · Shawn O. Pearce
Quoted 5 lines
Quoted 3 lines

Are you fetching from a configured remote that has tracking branches, or are you fetching through a one-shot URL pasted onto the command line?

Configured remote.

Hmm. I wonder if we should try to shortcut the commit walking in a case like this and just feed the tracking branches we already have.

You mean that the main culprit is the presense of thousdands of commits that fetcher has obtained through the other remotes (and his own) that the uploader makes fetcher walk all the way, in the false hope that there might be a commit among them that is closer to the commits being fetched than the ones at the tip of tracking branch the fetcher has for this uploader currently?

And the solution might be to tell only about the tips of remote tracking branches fetcher has obtained from this particular uploader, not about other remote tracking bracnesh it got from others or his own local branches (which may have merged from other remotes)?

It is a clever idea but I suspect it may not work well in practice. For example, suppose a project is two-tier, say, with top-level and subsystem repositories, the former of which regularly merge from the latter, and you are a participant primarily working on the subsystem. You fetch daily from the subsystem repository, but weekly from the top-level.

Now, when you fetch from the top-level, the remote tracking refs you have for it are much more stale than your other refs. The top-level would have acquired a lot more commits from the same subsystem repository since you fetched from there the last time, and you already have many of them through your daily fetch from the subsystem repository. To minimize the transfer in such a case, the fetcher does want to tell the uploader that it has those commits from the same subsystem repository, so that the commit walker can stop at a recent merge into the top-level from the subsystem repository.

There was a discussion about updating the commit walk exchange to bisect the history (skip and try a much older one to see if it is reachable, but to avoid overshooting, step back and see if a newer one is still common). It would be a lot more work and needs to be implemented as a new protocol capability, but I think it is the right way to go in the longer term.

#19 lore raw

Re: [PATCH] http/remote-curl: coddle picky servers

In reply to Mike Hommey

Hi,

Quoted 10 lines · Mike Hommey

Look closely at the start of the requested URL: /gitweb.cgi... It comes from this rule:

RewriteCond %{QUERY_STRING} ^(.+)$
RewriteRule ^/(.*)$ /gitweb.cgi$1 [L,PT]

which is global to the virtual host.

Anyways, while git.debian.org can certainly be fixed for that, other servers may want to do some different things with urls with parameters.

heh, I was suspecting some URL rewriting was going on.

Is this an issue that should be fixed in gitweb?

(added John 'Warthog9' Hawley to the Cc list, perhaps he might know.)

Signature · 2 lines
Cheers,
Ray Chuan
#20 lore raw

Re: [PATCH] http/remote-curl: coddle picky servers

Quoted 7 lines · Tay Ray Chuan
Quoted 6 lines · Mike Hommey

Look closely at the start of the requested URL: /gitweb.cgi... It comes from this rule:

RewriteCond %{QUERY_STRING} ^(.+)$
RewriteRule ^/(.*)$ /gitweb.cgi$1 [L,PT]

Quoted 8 lines
Quoted 4 lines

which is global to the virtual host.

Anyways, while git.debian.org can certainly be fixed for that, other servers may want to do some different things with urls with parameters.

heh, I was suspecting some URL rewriting was going on.

Is this an issue that should be fixed in gitweb?

I don't see why it should be. gitweb isn't a service CGI. I find it odd that someone would configure their website to route anything with a query string into gitweb. WTF?

Signature · 1 line
Shawn.
#21 lore raw

Re: [PATCH] http/remote-curl: coddle picky servers

Quoted 13 lines · Shawn O. Pearce
Quoted 12 lines · Tay Ray Chuan
Quoted 10 lines · Mike Hommey

Look closely at the start of the requested URL: /gitweb.cgi... It comes from this rule:

RewriteCond %{QUERY_STRING} ^(.+)$
RewriteRule ^/(.*)$ /gitweb.cgi$1 [L,PT]

which is global to the virtual host.

Anyways, while git.debian.org can certainly be fixed for that, other servers may want to do some different things with urls with parameters.

Quoted 7 lines
Quoted 3 lines

heh, I was suspecting some URL rewriting was going on.

Is this an issue that should be fixed in gitweb?

I don't see why it should be. gitweb isn't a service CGI. I find it odd that someone would configure their website to route anything with a query string into gitweb. WTF?

There was a good reason for it, but the implementation was too broad:
The main gitweb list (http://git.debian.org/) is made statically, because it is too long for gitweb to create it in a timely fashion.

So while the main page is made to be a static file, when the request has a query string, which means it's not the main gitweb list, the cgi is used.

Except this rule was also used for unrelated urls.

But as I said in my reply to Tay Ray Chuan, it's fixed.

Mike

#22 lore raw

Re: [PATCH] http/remote-curl: coddle picky servers

In reply to Tay Ray Chuan
Quoted 9 lines · Tay Ray Chuan

Hi,

Quoted 6 lines · Mike Hommey

Look closely at the start of the requested URL: /gitweb.cgi... It comes from this rule:

RewriteCond %{QUERY_STRING} ^(.+)$
RewriteRule ^/(.*)$ /gitweb.cgi$1 [L,PT]

Quoted 8 lines
Quoted 4 lines

which is global to the virtual host.

Anyways, while git.debian.org can certainly be fixed for that, other servers may want to do some different things with urls with parameters.

heh, I was suspecting some URL rewriting was going on.

Is this an issue that should be fixed in gitweb?

Nah, that's just an issue with the config at git.debian.org.

It's fixed already.

Mike

#23 lore raw
In reply to Yaroslav Halchenko

Hi,

Quoted 1 line · Yaroslav Halchenko

Cloning of the repository works fine with v1.6.5.7 but fails with v1.6.6-rc0.

this sounds like around the time the smart http protocol was introduced.

Quoted 4 lines

fatal: http://git.debian.org/git/pkg-exppsy/pymvpa.git/info/refs?service=git-upload-pack not found: did you run git update-server-info on the server?

on the server, 1.6.3.3 version of git was used to run git update-server-info.

hmm, are you using the WebDAV-flavour or the smart http protocol to host the repository?

Signature · 2 lines
Cheers,
Ray Chuan
#24 lore raw
In reply to Yaroslav Halchenko
Quoted 9 lines · Yaroslav Halchenko

$> GIT_TRACE=2 ./git clone http://git.debian.org/git/pkg-exppsy/pymvpa.git
trace: built-in: git 'clone' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git'
warning: templates not found /home/yoh/share/git-core/templates
Initialized empty Git repository in /home/yoh/proj/misc/git/pymvpa/.git/
trace: run_command: 'remote-curl' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git'
trace: exec: 'git' 'remote-curl' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git'
trace: exec: 'git-remote-curl' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git'
trace: run_command: 'git-remote-curl' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git'
fatal: http://git.debian.org/git/pkg-exppsy/pymvpa.git/info/refs?service=git-upload-pack not found: did you run git update-server-info on the server?

oh, and by the way, could you also run this again with GIT_CURL_VERBOSE=1?

Signature · 2 lines
Cheers,
Ray Chuan
#25 lore raw

$> GIT_CURL_VERBOSE=1 GIT_TRACE=2 ./git clone http://git.debian.org/git/pkg-exppsy/pymvpa.git
trace: built-in: git 'clone' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git'
warning: templates not found /home/yoh/share/git-core/templates
Initialized empty Git repository in /home/yoh/proj/misc/git/pymvpa/.git/
trace: run_command: 'remote-http' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git'
trace: exec: 'git' 'remote-http' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git'
trace: exec: 'git-remote-http' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git'
trace: run_command: 'git-remote-http' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git'
* Couldn't find host git.debian.org in the .netrc file; using defaults
* About to connect() to git.debian.org port 80 (#0)
* Trying 217.196.43.134... * Connected to git.debian.org (217.196.43.134) port 80 (#0)

Quoted 1 line

GET /git/pkg-exppsy/pymvpa.git/info/refs?service=git-upload-pack HTTP/1.1

User-Agent: git/1.6.6.267.g5b159
Host: git.debian.org
Accept: */*
Pragma: no-cache

* The requested URL returned error: 404
* Closing connection #0
fatal: http://git.debian.org/git/pkg-exppsy/pymvpa.git/info/refs?service=git-upload-pack not found: did you run git update-server-info on the server?

as for smart vs DAV -- don't see any smart alias handling in apache configuration (I have/had no clue about some smart http in git, just looked at apache template and saw smart aliases -- is there smth else to check within webserver config?)

Quoted 13 lines · Tay Ray Chuan
Quoted 9 lines · Yaroslav Halchenko

$> GIT_TRACE=2 ./git clone http://git.debian.org/git/pkg-exppsy/pymvpa.git
trace: built-in: git 'clone' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git'
warning: templates not found /home/yoh/share/git-core/templates
Initialized empty Git repository in /home/yoh/proj/misc/git/pymvpa/.git/
trace: run_command: 'remote-curl' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git'
trace: exec: 'git' 'remote-curl' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git'
trace: exec: 'git-remote-curl' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git'
trace: run_command: 'git-remote-curl' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git' 'http://git.debian.org/git/pkg-exppsy/pymvpa.git'
fatal: http://git.debian.org/git/pkg-exppsy/pymvpa.git/info/refs?service=git-upload-pack not found: did you run git update-server-info on the server?

oh, and by the way, could you also run this again with GIT_CURL_VERBOSE=1?

Signature · 5 lines
Yaroslav O. Halchenko
Postdoctoral Fellow,   Department of Psychological and Brain Sciences
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik        
#26 lore raw

Hi,

Quoted 10 lines · Yaroslav Halchenko

*   Trying 217.196.43.134... * Connected to git.debian.org (217.196.43.134) port 80 (#0)

Quoted 1 line

GET /git/pkg-exppsy/pymvpa.git/info/refs?service=git-upload-pack HTTP/1.1

User-Agent: git/1.6.6.267.g5b159
Host: git.debian.org
Accept: */*
Pragma: no-cache

* The requested URL returned error: 404
* Closing connection #0
fatal: http://git.debian.org/git/pkg-exppsy/pymvpa.git/info/refs?service=git-upload-pack not found: did you run git update-server-info on the server?

I don't think git's at fault here, as we're getting a 404 Not Found. Could you check that the repository (the one the url points to, after taking into any url rewriting) is a bare one, ie. has structure

pkg-exppsy/
|-pymvpa.git

    |-objects/
    |-info/
    |-refs/
    |-...

rather than the non-bare

pkg-exppsy/
|-pymvpa.git

    |-.git
      |-objects/
      |-info/
      |-refs/
      |-...

?

Quoted 4 lines

as for smart vs DAV -- don't see any smart alias handling in apache configuration (I have/had no clue about some smart http in git, just looked at apache template and saw smart aliases -- is there smth else to check within webserver config?)

If that's the case, I don't think it's related to your problem. (Btw, "smart" refers to the http protocol that git can use to sync your repo, via a CGI program on the server, instead of WebDAV. See git-http-backend(1) for details.)

Signature · 2 lines
Cheers,
Ray Chuan
#27 lore raw
Quoted 2 lines · Tay Ray Chuan
Quoted 1 line

fatal: http://git.debian.org/git/pkg-exppsy/pymvpa.git/info/refs?service=git-upload-pack not found: did you run git update-server-info on the server?

I don't think git's at fault here, as we're getting a 404 Not Found.

khe khe, pardon me, but if git can't talk to its nephew (ie its own repository which was created with earlier version), whenever another nephew (ie git of earlier version) can talk to it, I do consider it to be git's fault ;)

let me zoom in onto difference in communication between two different versions :

* Trying 217.196.43.134... * Connected to git.debian.org (217.196.43.134) port 80 (#0)

Quoted 1 line

GET /git/pkg-exppsy/pymvpa.git/info/refs HTTP/1.1

User-Agent: git/1.6.5
Host: git.debian.org
Accept: */*
Pragma: no-cache

< HTTP/1.1 200 OK

whenever, once again, for 1.6.6 it looked much shorter:

* Trying 217.196.43.134... * Connected to git.debian.org (217.196.43.134) port 80 (#0)

Quoted 1 line

GET /git/pkg-exppsy/pymvpa.git/info/refs?service=git-upload-pack HTTP/1.1

User-Agent: git/1.6.6.267.g5b159
Host: git.debian.org
Accept: */*
Pragma: no-cache

* The requested URL returned error: 404
* Closing connection #0
fatal: http://git.debian.org/git/pkg-exppsy/pymvpa.git/info/refs?service=git-upload-pack not found: did you run git update-server-info on the server?

Quoted 2 lines

Could you check that the repository (the one the url points to, after taking into any url rewriting) is a bare one, ie. has structure

yes - it is bare... I was 101% sure (isn't that a convention to have .git suffix for directories with bare repositories), but just to make sure:

$> cd /srv/git.debian.org/git/pkg-exppsy/pymvpa.git/ total 76
8 branches/ 8 config 8 description 8 HEAD 8 hooks/ 8 info/ 8 objects/ 12 packed-refs 8 refs/

Quoted 4 lines

If that's the case, I don't think it's related to your problem. (Btw, "smart" refers to the http protocol that git can use to sync your repo, via a CGI program on the server, instead of WebDAV. See git-http-backend(1) for details.)

thanks for info, I did know about this beastie.

I see no references to git-http-backend in apache config -- so indeed should not be the case... but from the git-http-backend description:

,---

| By default, only the `upload-pack` service is enabled, which serves
| 'git-fetch-pack' and 'git-ls-remote' clients, which are invoked from
| 'git-fetch', 'git-pull', and 'git-clone'.

`---

so, it looks like 1.6.6 for some reason decided to assume that it is "smart" http whenever it is not? is that the case here?

Signature · 5 lines
Yaroslav O. Halchenko
Postdoctoral Fellow,   Department of Psychological and Brain Sciences
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik        
#17 lore raw

Re: git fetch -v not at all verbose?

In reply to Junio C Hamano
Quoted 39 lines · Junio C Hamano
Quoted 8 lines · Shawn O. Pearce
Quoted 5 lines
Quoted 3 lines

Are you fetching from a configured remote that has tracking branches, or are you fetching through a one-shot URL pasted onto the command line?

Configured remote.

Hmm. I wonder if we should try to shortcut the commit walking in a case like this and just feed the tracking branches we already have.

You mean that the main culprit is the presense of thousdands of commits that fetcher has obtained through the other remotes (and his own) that the uploader makes fetcher walk all the way, in the false hope that there might be a commit among them that is closer to the commits being fetched than the ones at the tip of tracking branch the fetcher has for this uploader currently?

And the solution might be to tell only about the tips of remote tracking branches fetcher has obtained from this particular uploader, not about other remote tracking bracnesh it got from others or his own local branches (which may have merged from other remotes)?

It is a clever idea but I suspect it may not work well in practice. For example, suppose a project is two-tier, say, with top-level and subsystem repositories, the former of which regularly merge from the latter, and you are a participant primarily working on the subsystem. You fetch daily from the subsystem repository, but weekly from the top-level.

Now, when you fetch from the top-level, the remote tracking refs you have for it are much more stale than your other refs. The top-level would have acquired a lot more commits from the same subsystem repository since you fetched from there the last time, and you already have many of them through your daily fetch from the subsystem repository. To minimize the transfer in such a case, the fetcher does want to tell the uploader that it has those commits from the same subsystem repository, so that the commit walker can stop at a recent merge into the top-level from the subsystem repository.

Quoted 5 lines

There was a discussion about updating the commit walk exchange to bisect the history (skip and try a much older one to see if it is reachable, but to avoid overshooting, step back and see if a newer one is still common). It would be a lot more work and needs to be implemented as a new protocol capability, but I think it is the right way to go in the longer term.

I thought about this some more: it seems that nothing in pack-protocol.txt dictates that client has to send have lines in order. The whole logic would be on client side.

So a new capability will be there just in case we find a use for a server-side optimization later on, we don't need the client to behave differently in any way when this capability is enabled/disabled. Right?

Signature · 1 line
MST
#18 lore raw

Re: git fetch -v not at all verbose?

Quoted 11 lines · Michael S. Tsirkin

On Thu, Jan 21, 2010 at 09:42:36AM -0800, Junio C Hamano wrote:
...

Quoted 5 lines

There was a discussion about updating the commit walk exchange to bisect the history (skip and try a much older one to see if it is reachable, but to avoid overshooting, step back and see if a newer one is still common). It would be a lot more work and needs to be implemented as a new protocol capability, but I think it is the right way to go in the longer term.

I thought about this some more: it seems that nothing in pack-protocol.txt dictates that client has to send have lines in order. The whole logic would be on client side.

The current protocol may not require any order for it to function correctly in the sense that the sent pack will contain everything that is necessary, but it does require that commits on a lineage to be sent from near tip to near root if you want to have a _good_ common ancestor to be found.

If the downloader sends older commits first without sending some new ones, the uploader can say "Ok, I know about that old one you told me you have, so we could use that as a common commit" [*1*]. But there is no way for it to continue the sentence with "... but I cannot tell if other ones you told me you have that I know nothing about are all directly connected to that common one we just found (in which case that common one is the best we can do), or you have newer ones than the common commit that I also have but you omitted from the listing (in other words, if you didn't omit them, we could have found a better common commit). Could you please back up a bit and let us see if we can do better with newer ones?" with the current protocol exchange.

The downloader _could_, upon seeing an ACK to a commit that is an ancestor of commits that it skipped, try sending these skipped commits, without telling the uploader that it what it is doing. But the uploader will unilaterally decide when it thinks it has heard enough, after giving an ACK back in the original protocol, or after finding enough common ancestors to cover all the tips requested with WANTs, so I suspect that you may not have a chance to play such a game without an explicit protocol extension.

[Footnote]

*1* That is what an ACK means. In an multi-ack exchange, it also tells the downloader there is no point to give any ancestors of that commit, but allows the downloader to continue sending commits from other lineage.