crates.io data access policy
cover crates.io data access policy
It appears that the crate fetcher does not adhere to the data access policy of crates.io as is described here:
https://crates.io/data-access#api
Namely the part about adding a user-agent identifying the application and information on how to reach the project in case of issues.
Also users are requested to use the CDN instead of the download API, as that does not have a 1 request per second limit:
https://github.com/rust-lang/crates.io/issues/13482#issuecomment-4304855751
This patch addresses the request to use a CDN and seems to fix a recent breakage of the crate fetcher. It does not address the part about a user-agent since I do not know how to do that, and I do not know what to put in it anyway.
Johan Anderholm (1):
fetch2/crate: use CDN for fetching crates
lib/bb/fetch2/crate.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
Signature · 1 line
2.34.1
PATCH 1/1 fetch2/crate: use CDN for fetching crates
This avoids the 1 req/sec that the API has.
Reference: https://github.com/rust-lang/crates.io/issues/13482
Reference: https://blog.rust-lang.org/2024/03/11/crates-io-download-changes/
--- lib/bb/fetch2/crate.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
lib/bb/fetch2/crate.py +4 -1
@@ -68,8 +68,11 @@ class Crate(Wget): # if using upstream just fix it up nicely if host == 'crates.io': host = 'crates.io/api/v1/crates' + cdn_host = 'static.crates.io/crates' + else: + cdn_host = host - ud.url = "https://%s/%s/%s/download" % (host, name, version) + ud.url = "https://%s/%s/%s/download" % (cdn_host, name, version) ud.versionsurl = "https://%s/%s/versions" % (host, name) ud.parm['downloadfilename'] = "%s-%s.crate" % (name, version) if 'name' not in ud.parm:
Signature · 1 line
2.34.1
Hi Johan,
Quoted 10 lines · Johan Anderholm via lists.openembedded.org
[You don't often get email from [email protected]. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
It appears that the crate fetcher does not adhere to the data access policy of crates.io as is described here:
https://crates.io/data-access#api
Namely the part about adding a user-agent identifying the application and information on how to reach the project in case of issues.
Quoted 9 lines
Also users are requested to use the CDN instead of the download API, as that does not have a 1 request per second limit:
https://github.com/rust-lang/crates.io/issues/13482#issuecomment-4304855751
This patch addresses the request to use a CDN and seems to fix a recent breakage of the crate fetcher. It does not address the part about a user-agent since I do not know how to do that, and I do not know what to put in it anyway.
The crate fetcher is derived from the wget fetch method which does seem to implement a UA reporting bitbake and its version, see 234f9e810494 ("fetch2/wget: set User-Agent to 'bitbake/version' in checkstatus()"). The best thing would be to try yourself (I'm assuming having a webserver you control and change the fetcher to use your local webserver (or possibly even setting up a PREMIRRORS for crates)).
Cheers,
Quentin
Quoted 2 lines · Quentin Schulz via lists.openembedded.org
The crate fetcher is derived from the wget fetch method which does seem to implement a UA reporting bitbake and its version, see
234f9e810494("fetch2/wget: set User-Agent to 'bitbake/version' incheckstatus()"). The best thing would be to try yourself (I'm assuming having a webserver you control and change the fetcher to use your local webserver (or possibly even setting up a PREMIRRORS for crates)).
That’s checkstatus, not the actual fetch.
Put this on the list of reasons why I dislike the wget fetcher: two entirely distinct codepaths.
Ross