Design for offloading part of packfile response to CDN

There is a potential issue: a server which produces both the URIs and the packfile at roughly the same time (like the implementation in this patch set) will not have sideband access until it has concluded sending the URIs. Among other things, this means that the server cannot send keepalive packets until quite late in the response. One solution to this might be to add a feature that allows the server to use a sideband throughout the whole response - and this has other benefits too like allowing servers to inform the client throughout the whole fetch, not just at the end.

Subsequent patches will change how the output of pack-objects is processed, so extract that processing into its own function.

Currently, at most 1 character can be buffered (in the "buffered" local variable). One of those patches will require a larger buffer, so replace that "buffered" local variable with a buffer array.

Signed-off-by Jonathan Tan <[email protected]>

---
 Documentation/technical/packfile-uri.txt | 83 ++++++++++++++++++++++++
 Documentation/technical/protocol-v2.txt  |  6 +-
 2 files changed, 88 insertions(+), 1 deletion(-)
 create mode 100644 Documentation/technical/packfile-uri.txt

Documentation/technical/packfile-uri.txt new file +20 -0

new file mode 100644
index 0000000000..6535801486
@@ -0,0 +1,83 @@
+Packfile URIs
+=============
+
+This feature allows servers to serve part of their packfile response as URIs.
+This allows server designs that improve scalability in bandwidth and CPU usage
+(for example, by serving some data through a CDN), and (in the future) provides
+some measure of resumability to clients.
+
+This feature is available only in protocol version 2.
+
+Protocol
+--------
+
+The server advertises `packfile-uris`.
+
+If the client replies with the following arguments:
+
+ * packfile-uris
+ * thin-pack
+ * ofs-delta

Some thoughts here:

First, I'd like to see a section (and a bit in the implementation) requiring HTTPS if the original protocol is secure (SSH or HTTPS). Allowing the server to downgrade to HTTP, even by accident, would be a security problem.

Second, this feature likely should be opt-in for SSH. One issue I've seen repeatedly is that people don't want to use HTTPS to fetch things when they're using SSH for Git. Many people in corporate environments have proxies that break HTTP for non-browser use cases[0], and using SSH is the only way that they can make a functional Git connection.

Third, I think the server needs to be required to both support Range headers and never change the content of a URI, so that we can have resumable clone implicit in this design. There are some places in the world where connections are poor and fetching even the initial packfile at once might be a problem. (I've seen such questions on Stack Overflow, for example.)

Having said that, I think overall this is a good idea and I'm glad to see a proposal for it.

This is a partial implementation of upload-pack sending part of its packfile response as URIs.

The git command line expects Git servers to follow a specific order of

Documentation/technical/protocol-v2.txt +0 -0

     wanted-refs section
 	* This section is only included if the client has requested a
 	  ref using a 'want-ref' line and if a packfile section is also

Documentation/technical/packfile-uri.txt new file +4 -0

+Server design
+-------------
+
+The server can be trivially made compatible with the proposed protocol by

Documentation/technical/packfile-uri.txt new file +23 -0

+Future work
+-----------
+
+The protocol design allows some evolution of the server and client without any
+need for protocol changes, so only a small-scoped design is included here to
+form the MVP. For example, the following can be done:
+
+ * On the server, a long-running process that takes in entire requests and
+   outputs a list of URIs and the corresponding inclusion and exclusion sets of
+   objects. This allows, e.g., signed URIs to be used and packfiles for common
+   requests to be cached.
+ * On the client, resumption of clone. If a clone is interrupted, information
+   could be recorded in the repository's config and a "clone-resume" command
+   can resume the clone in progress. (Resumption of subsequent fetches is more
+   difficult because that must deal with the user wanting to use the repository
+   even after the fetch was interrupted.)
+
+There are some possible features that will require a change in protocol:
+
+ * Additional HTTP headers (e.g. authentication)
+ * Byte range support
+ * Different file formats referenced by URIs (e.g. raw object)
+

Documentation/technical/protocol-v2.txt +5 -1

index 345c00e08c..2cb1c41742 100644
@@ -313,7 +313,8 @@ header. Most sections are sent only when the packfile is sent.
 
     output = acknowledgements flush-pkt |
 	     [acknowledgments delim-pkt] [shallow-info delim-pkt]
-	     [wanted-refs delim-pkt] packfile flush-pkt
+	     [wanted-refs delim-pkt] [packfile-uris delim-pkt]
+	     packfile flush-pkt
 
     acknowledgments = PKT-LINE("acknowledgments" LF)
 		      (nak | *ack)

@@ -331,6 +332,9 @@ header. Most sections are sent only when the packfile is sent.
 		  *PKT-LINE(wanted-ref LF)
     wanted-ref = obj-id SP refname
 
+    packfile-uris = PKT-LINE("packfile-uris" LF) *packfile-uri
+    packfile-uri = PKT-LINE("uri" SP *%x20-ff LF)
+
     packfile = PKT-LINE("packfile" LF)
 	       *PKT-LINE(%x01-03 *%x00-ff)

So, this is OK, but

Documentation/technical/packfile-uri.txt line 30

Quoted 4 lines

Documentation/technical/packfile-uri.txt new file +4 -0

+Clients then should understand that the returned packfile could be incomplete,
+and that it needs to download all the given URIs before the fetch or clone is
+complete. Each URI should point to a Git packfile (which may be a thin pack and
+which may contain offset deltas).

weaken or remove the (parenthetical comment) in the last sentence, and replace the beginning of the section with something like

If the client replies with 'packfile-uris', when the server sends the packfile, it MAY send a packfile-uris section...

You may steal what I wrote in the above response to help the server-side folks to decide how to actually implement the "it MAY send a packfile-uris" part in the document.

static void create_pack_file(const struct object_array *have_obj,

upload-pack.c +2 -1

-			     const struct object_array *want_obj)
+			     const struct object_array *want_obj,
+			     int use_protocol_v2)
 {
 	struct child_process pack_objects = CHILD_PROCESS_INIT;
 	struct output_state output_state = {0};

@@ -260,9 +278,13 @@ static void create_pack_file(const struct object_array *have_obj,
 			 */
 			sz = xread(pack_objects.err, progress,

Quoted 1 line

The git command line expects Git servers to follow a specific order of

"Command line"? It sounds like you are talking about the order of command line arguments and options, but apparently that is not what you are doing. Is it "The git over-the-wire protocol"?

Earlier, we said that shallow-info is not given when packfile is not there. That is captured in the updated EBNF above. We don't have a corresponding removal of a bullet point for wanted-refs section below but probably that is because the original did not have corresponding bullet point to begin with.

* This section is only included if the client has requested a ref using a 'want-ref' line and if a packfile section is also included in the response.

Documentation/technical/packfile-uri.txt line 7

Quoted 4 lines

Documentation/technical/packfile-uri.txt new file +4 -0

+This feature allows servers to serve part of their packfile response as URIs.
+This allows server designs that improve scalability in bandwidth and CPU usage
+(for example, by serving some data through a CDN), and (in the future) provides
+some measure of resumability to clients.

Without reading the remainder, this makes readers anticipate a few good things ;-)

- "part of", so pre-generated constant material can be given from CDN and then followed-up by "filling the gaps" small packfile, perhaps?

- The "part of" transmission may not bring the repository up to date wrt to the "want" objects; would this feature involve "you asked history up to these commits, but with this pack-uri, you'll be getting history up to these (somewhat stale) commits"?

Documentation/technical/packfile-uri.txt line 20

Quoted 5 lines

Documentation/technical/packfile-uri.txt new file +5 -0

+If the client replies with the following arguments:
+
+ * packfile-uris
+ * thin-pack
+ * ofs-delta

"with the following" meaning "with all of the following", or "with any of the following"? Is there a reason why the server side must require that the client understands and is willing to accept a thin-pack when wanting to use packfile-uris? The same question for the ofs-delta.

My recommendation is to drop the mention of "thin" and "ofs" from the above list, and also from the following paragraph. The "it MAY send" will serve as a practical escape clause to allow a server/CDN implementation that *ALWAYS* prepares pregenerated material that can only be digested by clients that supports thin and ofs. Such a server can send packfile-URIs only when all of the three are given by the client and be compliant. And such an update to the proposed document would allow a more diskful server to prepare both thin and non-thin pregenerated packs and choose which one to give to the client depending on the capability.

Documentation/technical/packfile-uri.txt line 30

Quoted 4 lines

Documentation/technical/packfile-uri.txt new file +4 -0

+Clients then should understand that the returned packfile could be incomplete,
+and that it needs to download all the given URIs before the fetch or clone is
+complete. Each URI should point to a Git packfile (which may be a thin pack and
+which may contain offset deltas).

weaken or remove the (parenthetical comment) in the last sentence, and replace the beginning of the section with something like

If the client replies with 'packfile-uris', when the server sends the packfile, it MAY send a packfile-uris section...

You may steal what I wrote in the above response to help the server-side folks to decide how to actually implement the "it MAY send a packfile-uris" part in the document.

OK, this comes back to what I alluded to at the beginning. We could respond to a full-clone request by feeding a series of packfile-uris and some ref information, perhaps like this:

* Grab this packfile and update your remote-tracking refs and tags to these values; you'd be as if you cloned the project when it was at v1.0.

* When you are done with the above, grab this packfile and update your remote-tracking refs and tags to these values; you'd be as if you cloned the project when it was at v2.0.

* When you are done with the above, grab this packfile and update your remote-tracking refs and tags to these values; you'd be as if you cloned the project when it was at v3.0.

...

* When you are done with the above, here is the remaining packdata to bring you fully up to date with your original "want"s.

and before fully reading the proposal, I anticipated that it was what you were going to describe. The major difference is "up to the packdata given to you so far, you'd be as if you fetched these" ref information, which would allow you to be interrupted and then simply resume, without having to remember the set of packfile-uris yet to be processed across a fetch/clone failure. If you sucessfully fetch packfile for ..v1.0, you can update the remote-tracking refs to match as if you fetched back when that was the most recent state of the project, and then if you failed while transferring packfile for v1.0..v2.0, the resuming would just reissue "git fetch" internally.

Quoted 4 lines · Jonathan Tan

upload-pack.c +0 -0

@@ -126,6 +129,12 @@ static int read_pack_objects_stdout(int outfd, struct output_state *os)
 	}
 	os->used += readsz;
 

upload-pack.c static int read_pack_objects_stdout(int outfd, struct output_state *os) · line 135

Quoted 4 lines

upload-pack.c +4 -0

+	if (!os->packfile_started) {
+		os->packfile_started = 1;
+		if (use_protocol_v2)
+			packet_write_fmt(1, "packfile\n");

If we fix this function so that the only byte in the buffer is held back without emitted when os->used == 1 as I alluded to, this may have to be done a bit later, as with such a change, it is no longer guaranteed that send_client_data() will be called after this point.

Isn't progress output that goes to the channel #2 pretty much independent from the payload stream that carries the pkt-line command like "packfile" plus the raw pack stream? It somehow feels like an oxymoron to _buffer_ progress indicator, as it defeats the whole point of progress report to buffer it.

Quoted 6 lines

Quoted 4 lines · Jonathan Tan

upload-pack.c +0 -0

@@ -126,6 +129,12 @@ static int read_pack_objects_stdout(int outfd, struct output_state *os)
 	}
 	os->used += readsz;
 

Quoted 9 lines

upload-pack.c static int read_pack_objects_stdout(int outfd, struct output_state *os) · line 135

Quoted 4 lines

upload-pack.c +4 -0

+	if (!os->packfile_started) {
+		os->packfile_started = 1;
+		if (use_protocol_v2)
+			packet_write_fmt(1, "packfile\n");

If we fix this function so that the only byte in the buffer is held back without emitted when os->used == 1 as I alluded to, this may have to be done a bit later, as with such a change, it is no longer guaranteed that send_client_data() will be called after this point.

I'm not sure what you mean about there being no guarantee that send_client_data() is not called - in create_pack_file(), there is an "if (output_state.used > 0)" line (previously "if (0 <= buffered)") that outputs anything remaining.

... it might be
better if the server can send sideband throughout the whole response - perhaps that should be investigated first.

Quoted 5 lines

Quoted 1 line

The git command line expects Git servers to follow a specific order of

"Command line"? It sounds like you are talking about the order of command line arguments and options, but apparently that is not what you are doing. Is it "The git over-the-wire protocol"?

I meant to say the current Git implementation, as opposed to what is written in the specification. I'll replace it with "The current C Git implementation".

Quoted 5 lines

Earlier, we said that shallow-info is not given when packfile is not there. That is captured in the updated EBNF above. We don't have a corresponding removal of a bullet point for wanted-refs section below but probably that is because the original did not have corresponding bullet point to begin with.

That's because the corresponding bullet point had other information. Quoted in full below:

Quoted 3 lines

* This section is only included if the client has requested a ref using a 'want-ref' line and if a packfile section is also included in the response.

I could reword it to "If a packfile section is included in the response, this section is only included if the client has requested a ref using a 'want-ref' line", but I don't think that is significantly clearer.

Quoted 8 lines

Some thoughts here:

First, I'd like to see a section (and a bit in the implementation) requiring HTTPS if the original protocol is secure (SSH or HTTPS). Allowing the server to downgrade to HTTP, even by accident, would be a security problem.

Quoted 5 lines

Second, this feature likely should be opt-in for SSH. One issue I've seen repeatedly is that people don't want to use HTTPS to fetch things when they're using SSH for Git. Many people in corporate environments have proxies that break HTTP for non-browser use cases[0], and using SSH is the only way that they can make a functional Git connection.

Good points about SSH support and the client needing to control which protocols the server will send URIs for. I'll include a line in the client request in which the client can specify which protocols it is OK with.

Quoted 41 lines · Jonathan Tan

Signed-off-by Jonathan Tan <[email protected]>

---
 Documentation/technical/packfile-uri.txt | 83 ++++++++++++++++++++++++
 Documentation/technical/protocol-v2.txt  |  6 +-
 2 files changed, 88 insertions(+), 1 deletion(-)
 create mode 100644 Documentation/technical/packfile-uri.txt

Documentation/technical/packfile-uri.txt new file +28 -0

new file mode 100644
index 0000000000..6535801486
@@ -0,0 +1,83 @@
+Packfile URIs
+=============
+
+This feature allows servers to serve part of their packfile response as URIs.
+This allows server designs that improve scalability in bandwidth and CPU usage
+(for example, by serving some data through a CDN), and (in the future) provides
+some measure of resumability to clients.
+
+This feature is available only in protocol version 2.
+
+Protocol
+--------
+
+The server advertises `packfile-uris`.
+
+If the client replies with the following arguments:
+
+ * packfile-uris
+ * thin-pack
+ * ofs-delta
+
+when the server sends the packfile, it MAY send a `packfile-uris` section
+directly before the `packfile` section (right after `wanted-refs` if it is
+sent) containing HTTP(S) URIs. See protocol-v2.txt for the documentation of
+this section.
+
+Clients then should understand that the returned packfile could be incomplete,
+and that it needs to download all the given URIs before the fetch or clone is

Documentation/technical/packfile-uri.txt line 30

Quoted 2 lines

Documentation/technical/packfile-uri.txt new file +2 -0

+complete. Each URI should point to a Git packfile (which may be a thin pack and
+which may contain offset deltas).

First, I'd like to see a section (and a bit in the implementation) requiring HTTPS if the original protocol is secure (SSH or HTTPS). Allowing the server to downgrade to HTTP, even by accident, would be a security problem.

Second, this feature likely should be opt-in for SSH. One issue I've seen repeatedly is that people don't want to use HTTPS to fetch things when they're using SSH for Git. Many people in corporate environments have proxies that break HTTP for non-browser use cases[0], and using SSH is the only way that they can make a functional Git connection.

Third, I think the server needs to be required to both support Range headers and never change the content of a URI, so that we can have resumable clone implicit in this design. There are some places in the world where connections are poor and fetching even the initial packfile at once might be a problem. (I've seen such questions on Stack Overflow, for example.)

Having said that, I think overall this is a good idea and I'm glad to see a proposal for it.

[0] For example, a naughty-word filter may corrupt or block certain byte sequences that occur incidentally in the pack stream.

Documentation/technical/packfile-uri.txt new file +5 -0

+when the server sends the packfile, it MAY send a `packfile-uris` section
+directly before the `packfile` section (right after `wanted-refs` if it is
+sent) containing HTTP(S) URIs. See protocol-v2.txt for the documentation of
+this section.
+

Quoted 4 lines

Good points about SSH support and the client needing to control which protocols the server will send URIs for. I'll include a line in the client request in which the client can specify which protocols it is OK with.

What if a client is ok to fetch from some servers but not others (for example github.com and gitlab.com but nothing else)?

Or what if a client is ok to fetch using SSH from some servers and HTTPS from other servers but nothing else?

I also wonder in general how this would interact with promisor/partial clone remotes.

When we discussed promisor/partial clone remotes in the thread following this email:

https://public-inbox.org/git/[email protected]/

it looked like you were ok with having many promisor remotes, which I think could fill the same use cases especially related to large objects.

As clients would configure promisor remotes explicitly, there would be no issues about which protocol and servers are allowed or not.

If the issue is that you want the server to decide which promisor remotes would be used without the client having to do anything, maybe that could be something added on top of the possibility to have many promisor remotes.

Quoted 10 lines

Documentation/technical/packfile-uri.txt new file +9 -0

+when the server sends the packfile, it MAY send a `packfile-uris` section
+directly before the `packfile` section (right after `wanted-refs` if it is
+sent) containing HTTP(S) URIs. See protocol-v2.txt for the documentation of
+this section.
+
+Clients then should understand that the returned packfile could be incomplete,
+and that it needs to download all the given URIs before the fetch or clone is
+complete. Each URI should point to a Git packfile (which may be a thin pack and
+which may contain offset deltas).
[...]

Documentation/technical/packfile-uri.txt line 45

Quoted 5 lines

Documentation/technical/packfile-uri.txt new file +5 -0

+This is the implementation: a feature, marked experimental, that allows the
+server to be configured by one or more `uploadpack.blobPackfileUri=<sha1>
+<uri>` entries. Whenever the list of objects to be sent is assembled, a blob
+with the given sha1 can be replaced by the given URI. This allows, for example,
+servers to delegate serving of large blobs to CDNs.

Okey, so the server advertisement is not just "<urls>" but <oid><url> pairs. More on this later...

More concretely, I'd like to have a setup where a server can just dumbly point to some URL that probably has most of the data, without having any idea what OIDs are in it. So that e.g. some machine entirely disconnected from the server (and with just a regular clone) can continually generating an up-to-date-enough packfile.

I don't see how this is compatible with the server needing to send a bunch of "<oid> <url>" lines, or why a client "cannot declare that the fetch is complete until all URIs have been downloaded as packfiles". Can't it fall back on the normal dialog?

Other thoughts:

* If there isn't such a close coordination between git server & CDN, is there a case for having pack *.idx files on the CDN, so clients can inspect them to see if they'd like to download the full referenced pack?

* Without the server needing to know enough about the packs to advertise "<oid> <url>" is there a way to e.g. advertise 4x packs to clients:

big.pack, last-month.pack, last-week.pack, last-day.pack

Or some other optimistic negotiation where clients, even ones just doing regular fetches, can seek to get more up-to-date with one of the more recent packs before doing the first fetch in 3 days?

In the past I'd toyed with creating a similar "not quite CDN" setup using git-bundle.

Quoted 4 lines · brian m. carlson

First, I'd like to see a section (and a bit in the implementation) requiring HTTPS if the original protocol is secure (SSH or HTTPS). Allowing the server to downgrade to HTTP, even by accident, would be a security problem.

Maybe I've misunderstood the design (I'm writing some other follow-up E-Mails in this thread which might clarify things for me), but I don't see why.

We get the ref advertisement from the server. We don't need to trust the CDN server or the transport layer. We just download whatever we get from there, validate the packfile with SHA-1 (and in the future SHA-256). It doesn't matter if the CDN transport is insecure.

You can do this offline with git today, you don't need to trust me to trust that my copy of git.git I give you on a sketchy USB stick is genuine. Just unpack it, then compare the SHA-1s you get with:

git ls-remote https://github.com/git/git.git

So this is a case similar to Debian's where they distribute packages over http, but manifests over https: https://whydoesaptnotusehttps.com

Quoted 5 lines

Second, this feature likely should be opt-in for SSH. One issue I've seen repeatedly is that people don't want to use HTTPS to fetch things when they're using SSH for Git. Many people in corporate environments have proxies that break HTTP for non-browser use cases[0], and using SSH is the only way that they can make a functional Git connection.

Yeah, there should definitely be accommodations for such clients, per my reading clients can always ignore the CDN and proceed with a normal negotiation. Isn't that enough, or is something extra needed?

Quoted 16 lines · Ævar Arnfjörð Bjarmason

Quoted 4 lines · brian m. carlson

First, I'd like to see a section (and a bit in the implementation) requiring HTTPS if the original protocol is secure (SSH or HTTPS). Allowing the server to downgrade to HTTP, even by accident, would be a security problem.

Maybe I've misunderstood the design (I'm writing some other follow-up E-Mails in this thread which might clarify things for me), but I don't see why.

We get the ref advertisement from the server. We don't need to trust the CDN server or the transport layer. We just download whatever we get from there, validate the packfile with SHA-1 (and in the future SHA-256). It doesn't matter if the CDN transport is insecure.

Quoted 8 lines

You can do this offline with git today, you don't need to trust me to trust that my copy of git.git I give you on a sketchy USB stick is genuine. Just unpack it, then compare the SHA-1s you get with:

git ls-remote https://github.com/git/git.git

So this is a case similar to Debian's where they distribute packages over http, but manifests over https: https://whydoesaptnotusehttps.com

This assumes that integrity of the data is the only reason you'd want to use HTTPS. There's also confidentiality. Perhaps a user is downloading data that will help them circumvent the Great Firewall of China. A downgrade to HTTP could result in a long prison sentence.

Furthermore, some ISPs tamper with headers to allow tracking, and some environments (e.g. schools and libraries) perform opportunistic filtering on HTTP connections to filter certain content (and a lot of this filtering is really simplistic).

Moreover, Google is planning on using this and filters in place of Git LFS for large objects. I expect that if this approach becomes viable, it may actually grow authentication functionality, or, depending on how the series uses the existing code, it may already have it. In such a case, we should not allow authentication to go over a plaintext connection when the user thinks that the connection they're using is encrypted (since they used an SSH or HTTPS URL to clone or fetch).

Downgrades from HTTPS to HTTP are generally considered CVE-worthy. We need to make sure that we refuse to allow a downgrade on the client side, even if the server ignores our request for a secure protocol.

Quoted 6 lines

Quoted 5 lines

Second, this feature likely should be opt-in for SSH. One issue I've seen repeatedly is that people don't want to use HTTPS to fetch things when they're using SSH for Git. Many people in corporate environments have proxies that break HTTP for non-browser use cases[0], and using SSH is the only way that they can make a functional Git connection.

Quoted 3 lines

Yeah, there should definitely be accommodations for such clients, per my reading clients can always ignore the CDN and proceed with a normal negotiation. Isn't that enough, or is something extra needed?

I think at least a config option and a command line flag are needed to be able to turn CDN usage off. There needs to be an easy way for people in broken environments to circumvent the breakage.

Quoted 9 lines

Quoted 5 lines

Good points about SSH support and the client needing to control which protocols the server will send URIs for. I'll include a line in the client request in which the client can specify which protocols it is OK with.

What if a client is ok to fetch from some servers but not others (for example github.com and gitlab.com but nothing else)?

Quoted 2 lines

Or what if a client is ok to fetch using SSH from some servers and HTTPS from other servers but nothing else?

The objects received from the various CDNs are still rehashed by the client (so they are identified with the correct name), and if the client is fetching from a server, presumably it can trust the URLs it receives (just like it trusts ref names, and so on). Do you know of a specific case in which a client wants to fetch from some servers but not others?

(In any case, if this happens, the client can just disable the CDN support.)

Quoted 15 lines

I also wonder in general how this would interact with promisor/partial clone remotes.

When we discussed promisor/partial clone remotes in the thread following this email:

https://public-inbox.org/git/[email protected]/

it looked like you were ok with having many promisor remotes, which I think could fill the same use cases especially related to large objects.

As clients would configure promisor remotes explicitly, there would be no issues about which protocol and servers are allowed or not.

Quoted 4 lines

If the issue is that you want the server to decide which promisor remotes would be used without the client having to do anything, maybe that could be something added on top of the possibility to have many promisor remotes.

It's true that there is a slight overlap with respect to large objects, but this protocol can also handle large sets of objects being offloaded to CDN, not only single ones.

(The included implementation only handles
single objects, as a minimum viable product, but it is conceivable that the server implementation is later expanded to allow offloading of sets of objects.)

And this protocol is meant to be able to use CDNs to help serve objects, whether single objects or sets of objects. In the case of promisor remotes, the thing we fetch from has to be a Git server.

(We could use
dumb HTTP from a CDN, but that defeats the purpose in at least one way - with dumb HTTP, we have to fetch objects individually, but with URL support, we can fetch objects as sets too.)