Skip to contentSkip to conversation

crates.io data access policy

PATCH In review Johan Anderholm 3 participants 4 messages Apr 27

lore
Share & tools
lore
#0 lore raw

cover crates.io data access policy

It appears that the crate fetcher does not adhere to the data access policy of crates.io as is described here:

https://crates.io/data-access#api

Namely the part about adding a user-agent identifying the application and information on how to reach the project in case of issues.

Also users are requested to use the CDN instead of the download API, as that does not have a 1 request per second limit:

https://github.com/rust-lang/crates.io/issues/13482#issuecomment-4304855751

This patch addresses the request to use a CDN and seems to fix a recent breakage of the crate fetcher. It does not address the part about a user-agent since I do not know how to do that, and I do not know what to put in it anyway.

Johan Anderholm (1):
fetch2/crate: use CDN for fetching crates

 lib/bb/fetch2/crate.py | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
Signature · 1 line
2.34.1
#1 lore raw

PATCH 1/1 fetch2/crate: use CDN for fetching crates

This avoids the 1 req/sec that the API has.

Reference: https://github.com/rust-lang/crates.io/issues/13482
Reference: https://blog.rust-lang.org/2024/03/11/crates-io-download-changes/

---
 lib/bb/fetch2/crate.py | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
lib/bb/fetch2/crate.py +4 -1
index e611736f0..2d3078899 100644
@@ -68,8 +68,11 @@ class Crate(Wget):
         # if using upstream just fix it up nicely
         if host == 'crates.io':
             host = 'crates.io/api/v1/crates'
+            cdn_host = 'static.crates.io/crates'
+        else:
+            cdn_host = host
 
-        ud.url = "https://%s/%s/%s/download" % (host, name, version)
+        ud.url = "https://%s/%s/%s/download" % (cdn_host, name, version)
         ud.versionsurl = "https://%s/%s/versions" % (host, name)
         ud.parm['downloadfilename'] = "%s-%s.crate" % (name, version)
         if 'name' not in ud.parm:
Signature · 1 line
2.34.1
#2 lore raw
In reply to Johan Anderholm

Hi Johan,

Quoted 10 lines · Johan Anderholm via lists.openembedded.org

[You don't often get email from [email protected]. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]

It appears that the crate fetcher does not adhere to the data access policy of crates.io as is described here:

https://crates.io/data-access#api

Namely the part about adding a user-agent identifying the application and information on how to reach the project in case of issues.

Quoted 9 lines

Also users are requested to use the CDN instead of the download API, as that does not have a 1 request per second limit:

https://github.com/rust-lang/crates.io/issues/13482#issuecomment-4304855751

This patch addresses the request to use a CDN and seems to fix a recent breakage of the crate fetcher. It does not address the part about a user-agent since I do not know how to do that, and I do not know what to put in it anyway.

The crate fetcher is derived from the wget fetch method which does seem to implement a UA reporting bitbake and its version, see 234f9e810494 ("fetch2/wget: set User-Agent to 'bitbake/version' in checkstatus()"). The best thing would be to try yourself (I'm assuming having a webserver you control and change the fetcher to use your local webserver (or possibly even setting up a PREMIRRORS for crates)).

Cheers,
Quentin

#3 lore raw
Quoted 2 lines · Quentin Schulz via lists.openembedded.org

The crate fetcher is derived from the wget fetch method which does seem to implement a UA reporting bitbake and its version, see 234f9e810494 ("fetch2/wget: set User-Agent to 'bitbake/version' in checkstatus()"). The best thing would be to try yourself (I'm assuming having a webserver you control and change the fetcher to use your local webserver (or possibly even setting up a PREMIRRORS for crates)).

That’s checkstatus, not the actual fetch.

Put this on the list of reasons why I dislike the wget fetcher: two entirely distinct codepaths.

Ross