http-fetch enhancements
cover http-fetch enhancements
The following series contains some http-fetch enhancements, based on our requirements for use of SSL client certificates and partial HTTP transfers.
Signature · 2 lines
For a successful technology, reality must take precedence over public relations, for nature cannot be fooled.
Quoted 3 lines · Nick Hengeveld
The following series contains some http-fetch enhancements, based on our requirements for use of SSL client certificates and partial HTTP transfers.
If you happen to know how to have curl do multiple simultaneous downloads, that would be a big performance win, and I should be able to explain how to get this to work. I haven't gotten around to learning libcurl well enough to do the flow control.
-Daniel
*This .sig left intentionally blank*
Quoted 4 lines · Daniel Barkalow
If you happen to know how to have curl do multiple simultaneous downloads, that would be a big performance win, and I should be able to explain how to get this to work. I haven't gotten around to learning libcurl well enough to do the flow control.
The curl multi interface looks pretty straightforward. What did you have in mind as far as which requests would be running concurrently and how they would need to be limited?
Signature · 2 lines
For a successful technology, reality must take precedence over public relations, for nature cannot be fooled.
Quoted 10 lines · Nick Hengeveld
Quoted 4 lines · Daniel Barkalow
If you happen to know how to have curl do multiple simultaneous downloads, that would be a big performance win, and I should be able to explain how to get this to work. I haven't gotten around to learning libcurl well enough to do the flow control.
The curl multi interface looks pretty straightforward. What did you have in mind as far as which requests would be running concurrently and how they would need to be limited?
The way fetch.c calls the functions, there's a prefetch() that indicates that a given object is needed, and a fetch() that is responsible for making sure the object is available when it returns (or returning an error). It is arranged such that the same list of hashes is given to each of the functions in the same order.
One method is to send requests in prefetch() and accept responses in fetch(); this is what git-ssh-pull now does. This seems in practice to lead to ~100 outstanding requests at the high point, which is great for throughput, but I'm not sure how polite it is. IIRC, browsers tend to do ~4 simultaneous connections, or at least used to.
The other method is to keep track of what you're fetching, and block in prefetch() if too many connections are in use until some connections are free, or in fetch() if that is called before the download is complete.
Note that it should theoretically be possible to make additional requests on the same connection, provided it's "keep alive", even before reading the response, so long as the code is able to figure out what happened if the server actually closes it (after the first request) instead of serving the later ones.
-Daniel
*This .sig left intentionally blank*