gitweb.cgi
It is increasingly clear that gitweb.cgi is producing an unacceptable load on the kernel.org servers. Most of the hits we get are either the gitweb front page or the gitweb rss feeds, and it's eating I/O bandwidth like crazy.
This has become particularly painful during the current one-server outage.
Kay, gitweb really needs to be able to do caching, or be run behind a caching proxy. Otherwise I will have to turn it off until we can come up with a dedicated piece of server hardware for it.
-hpa
Quoted 2 lines · H. Peter Anvin
It is increasingly clear that gitweb.cgi is producing an unacceptable load on the kernel.org servers.
Sure, sorry, was on 3 conferences in a row the last weeks.
Quoted 3 lines
Most of the hits we get are either the
gitweb front page or the gitweb rss feeds, and it's eating I/O bandwidth like crazy.
I tested some stuff on these boxes and 30 stat() calls alone take app. 2 seconds on these boxes cause of I/O load ... :)
Quoted 5 lines
This has become particularly painful during the current one-server outage.
Kay, gitweb really needs to be able to do caching, or be run behind a caching proxy. Otherwise I will have to turn it off until we can come up with a dedicated piece of server hardware for it.
How about Apache's mod_cache? Worked nicely for me several times in other setups.
Kay
Quoted 8 lines · Kay Sievers
Quoted 4 lines
This has become particularly painful during the current one-server outage.
Kay, gitweb really needs to be able to do caching, or be run behind a caching proxy. Otherwise I will have to turn it off until we can come up with a dedicated piece of server hardware for it.
How about Apache's
mod_cache? Worked nicely for me several times in other setups.
I will look at it and see if I can make it work properly.
-hpa
Quoted 6 lines · Kay Sievers
Quoted 2 lines
Most of the hits we get are either the gitweb front page or the gitweb rss feeds, and it's eating I/O bandwidth like crazy.
I tested some stuff on these boxes and 30
stat()calls alone take app. 2 seconds on these boxes cause of I/O load ... :)
Welcome to my hell :)
I set up mod_cache (which I didn't know about, silly me) and so far it seems to work and has produced a tremendous decrease in load and improvement in response time. I do, have, however, a request. There are some gitweb pages which are more likely to change than others; in particular, some gitweb pages will *never* change (because they directly reflect immutable git data.)
If gitweb could produce Last-Modified and Expires headers where appropriate, it should improve caching performance.
-hpa
Quoted 6 lines · H. Peter Anvin
Quoted 5 lines · Kay Sievers
Quoted 4 lines
Most of the hits we get are either the
gitweb front page or the gitweb rss feeds, and it's eating I/O bandwidth like crazy.
Quoted 6 lines
Quoted 4 lines
I tested some stuff on these boxes and 30
stat()calls alone take app. 2 seconds
on these boxes cause of I/O load ... :)Welcome to my hell :)
Yeah, I get an idea now :)
Quoted 3 lines
I set up
mod_cache(which I didn't know about, silly me) and so far it seems to work and has produced a tremendous decrease in load and improvement in response time.
Great! Hope that will work.
Quoted 4 lines
I do, have, however, a request. There
are some gitweb pages which are more likely to change than others; in particular, some gitweb pages will *never* change (because they directly reflect immutable git data.)
Yes, makes sense.
Quoted 2 lines
If gitweb could produce Last-Modified and Expires headers where appropriate, it should improve caching performance.
I've added the Expires: header to the commit and commitdiff pages with one whole day ahead. Let's see if that will help...
Kay
Quoted 10 lines · H. Peter Anvin
Quoted 7 lines · Kay Sievers
Quoted 2 lines
Most of the hits we get are either the gitweb front page or the gitweb rss feeds, and it's eating I/O bandwidth like crazy.
I tested some stuff on these boxes and 30
stat()calls alone take app. 2 seconds on these boxes cause of I/O load ... :)
Quoted 7 lines
Welcome to my hell :)
I set up
mod_cache(which I didn't know about, silly me) and so far it seems to work and has produced a tremendous decrease in load and improvement in response time. I do, have, however, a request. There are some gitweb pages which are more likely to change than others; in particular, some gitweb pages will *never* change (because they directly reflect immutable git data.)If gitweb could produce Last-Modified and Expires headers where appropriate, it should improve caching performance.
-hpa
Some other areas for improvement would be to seperate out the git icon and the style sheet into seperate static files.
--
Brian Gerst
Quoted 4 lines · H. Peter Anvin
I set up
mod_cache(which I didn't know about, silly me) and so far it seems to work and has produced a tremendous decrease in load and improvement in response time.
It really doesn't work very well with the "front page" though..
Doing a "Save page as.." shows that it's not a huge page: it's roughly 700 lines long and 57kB in size, but pressing the reload button (or just going somewhere else and coming back immediately) takes 45 seconds to reload for me.
Trying again shows that it _is_ cached if you press the reload button immediately again, but I haven't quite figured out how long the cache timeout is. It seems to be around one minute (from some very preliminary tests it's more than 25 seconds, but less than a minute and a half).
Considering that apparently the load is enough that it takes 45 seconds to generate (scary in itself), is should clearly be cached for more than one minute. More like ten minutes or half an hour, especially since mirroring any content changes takes longer than that anyway.
Now, I suspect all the content on kernel.org could easily be cached for ten minutes.
As far as I can tell, mod_cache without any expiry information uses
CacheDefaultExpire
which should default to one hour according to the docs. Have you changed that to one minute? Maybe making it 10 minutes would be better?
That said, I tried to figure out how the front page is generated, but haven't quite. Can somebody (Kay?) please say what it does most, and I can try to make sure git does that efficiently..
Linus
Quoted 7 lines · Linus Torvalds
It really doesn't work very well with the "front page" though..
Doing a "Save page as.." shows that it's not a huge page: it's roughly 700 lines long and 57kB in size, but pressing the reload button (or just going somewhere else and coming back immediately) takes 45 seconds to reload for me.
Trying again shows that it _is_ cached if you press the reload button immediately again, but I haven't quite figured out how long the cache timeout is. It seems to be around one minute (from some very preliminary tests it's more than 25 seconds, but less than a minute and a half).
The cache timeout is set to 300 seconds, however, that's per server, of course.
Quoted 1 line
Considering that apparently the load is enough that it takes 45 seconds to generate (scary in itself), is should clearly be cached for more than one minute. More like ten minutes or half an hour, especially since mirroring any content changes takes longer than that anyway.
The latency for an I/O operation on the kernel.org servers is positively scary.
-hpa
Optimize common case of git-rev-list (was Re: gitweb.cgi)
Quoted 8 lines · H. Peter Anvin
Quoted 5 lines
Considering that apparently the load is enough that it takes 45 seconds to generate (scary in itself), is should clearly be cached for more than one minute. More like ten minutes or half an hour, especially since mirroring any content changes takes longer than that anyway.
The latency for an I/O operation on the kernel.org servers is positively scary.
I took a look at webgit, and it looks like at least for the "projects" page, the most common operation ends up being basically
git-rev-list --header --parents --max-count=1 HEAD
Now, the thing is, the way "git-rev-list" works, it always keeps on popping the parents and parsing them in order to build the list of parents, and it turns out that even though we just want a single commit, git-rev-list will invariably look up _three_ generations of commits.
It will parse:
- the commit we want (it obviously needs this)
- it's parent(s) as part of the "pop_most_recent_commit()" logic
- it will then pop one of the parents before it notices that it doesn't need any more
- and as part of popping the parent, it will parse the grandparent (again due to "pop_most_recent_commit()".
Now, I've strace'd it, and it really is pretty efficient on the whole, but if things aren't nicely cached, and with long-latency IO, doing those two extra objects (at a minimum - if the parent is a merge it will be more) is just wasted time, and potentially a lot of it.
So here's a quick special-case for the trivial case of "just one commit, and no date-limits or other special rules".
---
I've actually tried to test it (but hey, "exhaustive" is hard with all the different options), and I tried to be very careful to only do the special-case when really nothing can go wrong, and this all looks obvious.
But buyer beware.
Btw, doing an "strace" on git-rev-list shows that most of the system calls by far are the dynamic loader. Now, those accesses _should_ all be cached, so they should be fast and low-latency, but it's entirely possible that for a server configuration you might want to actually link things statically. Or not. Just a thought.
rev-list.c +4 -0
@@ -624,6 +624,10 @@ int main(int argc, char **argv)if (!merge_order) {sort_by_date (&list); +if (list && !limited && max_count ==1 ) { +show_commit (list->item); +return 0 ; + }if (limited) list =limit_list (list);if (topo_order)
Quoted 7 lines · Linus Torvalds
It really doesn't work very well with the "front page" though..
Doing a "Save page as.." shows that it's not a huge page: it's roughly 700 lines long and 57kB in size, but pressing the reload button (or just going somewhere else and coming back immediately) takes 45 seconds to reload for me.
Trying again shows that it _is_ cached if you press the reload button immediately again, but I haven't quite figured out how long the cache timeout is. It seems to be around one minute (from some very preliminary tests it's more than 25 seconds, but less than a minute and a half).
It turns out that the default CacheSize is only 256K. D'oh! Fixed.
I also changed the CacheDefaultExpire to 600 seconds.
Quoted 1 line
That said, I tried to figure out how the front page is generated, but haven't quite. Can somebody (Kay?) please say what it does most, and I can try to make sure git does that efficiently..
The only thing the front page really should need is to know when the last change to the tree was, which presumably means looking at each head of each tree and follow the chain until there is a datable object.
-hpa
Quoted 4 lines · H. Peter Anvin
It turns out that the default CacheSize is only 256K. D'oh! Fixed.
I also changed the CacheDefaultExpire to 600 seconds.
Ok, that sounds like it should improve things. My quick tests didn't seem to show any difference, though. Do you need to re-load the apache module or something?
Quoted 3 lines
The only thing the front page really should need is to know when the last change to the tree was, which presumably means looking at each head of each tree and follow the chain until there is a datable object.
Yeah. I tried to follow gitweb.cgi, but I'm neither http- nor perl-literate, so I'm not sure I caught everything.
But it does seem to basically end up doing a "git_read_commit()" for each project, and that in turn was doing the "git-rev-list --max-count=1" thing that I just sent out a suggested improvement for.
It effectively removes two or more copies of
stat64("/objects/xy/zzy", {...})
fd = open("objects/xy/zzy", O_RDONLY|O_NOATIME) addr = mmap(NULL, size, PROT_READ, MAP_PRIVATE, fd, 0) close(fd)
munmap(addr, size)
which really should be very cheap operations, but hey, if the disk head is somewhere else (and busy) and it's not cached, it can be quite expensive. Especially since we don't end up usign the result.
I'm sure there's room for improvement inside gitweb itself too, but maybe the git-rev-list optimization will help.
Linus
Quoted 31 lines · Linus Torvalds
Quoted 4 lines · H. Peter Anvin
It turns out that the default CacheSize is only 256K. D'oh! Fixed.
I also changed the CacheDefaultExpire to 600 seconds.
Ok, that sounds like it should improve things. My quick tests didn't seem to show any difference, though. Do you need to re-load the apache module or something?
Quoted 3 lines
The only thing the front page really should need is to know when the last change to the tree was, which presumably means looking at each head of each tree and follow the chain until there is a datable object.
Yeah. I tried to follow gitweb.cgi, but I'm neither http- nor perl-literate, so I'm not sure I caught everything.
But it does seem to basically end up doing a "
git_read_commit()" for each project, and that in turn was doing the "git-rev-list --max-count=1" thing that I just sent out a suggested improvement for.It effectively removes two or more copies of
stat64("/objects/xy/zzy", {...})
fd =open("objects/xy/zzy", O_RDONLY|O_NOATIME)addr =mmap(NULL, size, PROT_READ, MAP_PRIVATE, fd, 0)close(fd)
munmap(addr, size)
Quoted 6 lines
which really should be very cheap operations, but hey, if the disk head is somewhere else (and busy) and it's not cached, it can be quite expensive. Especially since we don't end up usign the result.
I'm sure there's room for improvement inside gitweb itself too, but maybe the git-rev-list optimization will help.
There definitely is! But I tried a single "stat() all HEAD files" with a simple script and it took more than 3 seconds for the 80 trees. Then I gave up "optimizing" and was sure we want to have a single-file cached front page instead. :)
Kay
Quoted 10 lines · Linus Torvalds
Quoted 3 lines · H. Peter Anvin
It turns out that the default CacheSize is only 256K. D'oh! Fixed.
I also changed the CacheDefaultExpire to 600 seconds.
Ok, that sounds like it should improve things. My quick tests didn't seem to show any difference, though. Do you need to re-load the apache module or something?
Yes, but I did that. It seems very strange when something hits the cache. A cgi script can apparently be run quite a few number of times before mod_cache sees it globally.
-hpa
Quoted 4 lines · H. Peter Anvin
Yes, but I did that. It seems very strange when something hits the cache. A cgi script can apparently be run quite a few number of times before
mod_cachesees it globally.
Well, I tried again this morning, and _boy_ is it better. The "projects" thing came up immediately, no waiting.
Maybe it just took a while for the mod_cache thing to take effect as Apache swarm members time out?
Or maybe it's just that the west coast is really quiet at 7AM.
Linus
Quoted 5 lines · Linus Torvalds
Well, I tried again this morning, and _boy_ is it better. The "projects" thing came up immediately, no waiting.
Maybe it just took a while for the
mod_cachething to take effect as Apache swarm members time out?
No, that's not it. It takes killing httpd and restarting it. Rather, it looks like mod_cache in different swarm members doesn't always communicate instantly, which is also underscored by the fact that cache files don't appear into the filesystem until after a while.
-hpa