Skip to contentSkip to conversation

crate.io data access policy (user agent)

PATCH New Johan Anderholm 1 participant 3 messages Apr 28

lore
Share & tools
lore
#1 lore raw

PATCH 1/2 fetch2/wget: allow identifying with a bitbake user agent

In reply to Johan Anderholm

Certain hosts do not allow identifying as wget, but want an application specific user agent.

This change causes a slight change in how checkstatus will identify. Instead of simply using bitbake/<version>, it will now also include project contact information:

bitbake/<version> (https://lists.openembedded.org/g/bitbake-devel; [email protected])

This change was made to comply with the data access policy of crates.io, while not introducing too many user agent variants.

Some more information on user agents can be found in 234f9e810494

---
 lib/bb/fetch2/wget.py | 19 ++++++++++++++++++-
 1 file changed, 18 insertions(+), 1 deletion(-)
lib/bb/fetch2/wget.py +18 -1
index 6ac4306c0..93c8b73f4 100644
@@ -60,6 +60,19 @@ class Wget(FetchMethod):
         """
         return (d.getVar("BB_CHECK_SSL_CERTS") or "1") != "0"
 
+    def identify_as_bitbake(self):
+        """
+        Should wget identify as bitbake?
+        """
+        return False
+
+    def bitbake_user_agent(self):
+        """
+        Certain hosts require users to identify with an application specific user-agent.
+        E.g. crates.io according to https://crates.io/data-access
+        """
+        return "bitbake/{} (https://lists.openembedded.org/g/bitbake-devel; [email protected])".format(bb.__version__)
+
     def supports(self, ud, d):
         """
         Check to see if a given url can be fetched with wget.
@@ -122,6 +135,10 @@ class Wget(FetchMethod):
                 # Authorization header.
                 fetchcmd += " --user=%s --password=%s" % (ud.user, ud.pswd)
 
+        # Some hosts, e.g. crates.io, may require using a special user-agent.
+        if self.identify_as_bitbake():
+            fetchcmd += " --user-agent='%s'" % self.bitbake_user_agent()
+
         uri = ud.url.split(";")[0]
         fetchcmd += " --continue --directory-prefix=%s '%s'" % (dldir, uri)
         self._runwget(ud, d, fetchcmd, False)
@@ -371,7 +388,7 @@ class Wget(FetchMethod):
                 # Some servers (FusionForge, as used on Alioth) require that the
                 # optional Accept header is set.
                 r.add_header("Accept", "*/*")
-                r.add_header("User-Agent", "bitbake/{}".format(bb.__version__))
+                r.add_header("User-Agent", self.bitbake_user_agent())
                 def add_basic_auth(login_str, request):
                     '''Adds Basic auth to http request, pass in login:password as string'''
                     import base64
Signature · 1 line
2.34.1
#0 lore raw

cover crate.io data access policy (user agent)

These patches addresses the part about setting an application specific user agent when making API calls to crates.io:

https://crates.io/data-access#api

I'm a little bit hesitant on the first patch, as that changes how checkstatus identifies slightly by adding contact information. Another option would be to leave it intact except for the crate fetcher. But I feel this is introducing more complexity in favor of, quite possibly excessive, defensiveness.

A third option is to leave out the contact information altogether and just use bitbake/<version> since the request about contact information is only "strongly suggested".

Johan Anderholm (2):
fetch2/wget: allow identifying with a bitbake user agent fetch2/crate: identify as bitbake to crates.io

 lib/bb/fetch2/crate.py | 18 +++++++++++++++++-
 lib/bb/fetch2/wget.py  | 19 ++++++++++++++++++-
 2 files changed, 35 insertions(+), 2 deletions(-)
Signature · 1 line
2.34.1
#2 lore raw

PATCH 2/2 fetch2/crate: identify as bitbake to crates.io

In order to adhere to the data access policy of crates.io, a user-agent header that identifies the application must be used.

Reference: https://github.com/rust-lang/crates.io/issues/13482
Reference: https://crates.io/data-access

---
 lib/bb/fetch2/crate.py | 18 +++++++++++++++++-
 1 file changed, 17 insertions(+), 1 deletion(-)
lib/bb/fetch2/crate.py +17 -1
index 2d3078899..5b6c715a3 100644
@@ -22,10 +22,25 @@ from   bb.fetch2.wget import Wget
 class Crate(Wget):
 
     """Class to fetch crates via wget"""
+    using_crates_io = False
 
     def _cargo_bitbake_path(self, rootdir):
         return os.path.join(rootdir, "cargo_home", "bitbake")
 
+    def identify_as_bitbake(self):
+        """
+        Should wget identify as bitbake?
+        """
+        # In order to comply with the data access policy of crates.io, a
+        # user-agent with application and contact information should be used:
+        #
+        # https://crates.io/data-access
+        if self.using_crates_io:
+            return True
+        else:
+            # Default to whatever Wget think is proper
+            return super(Crate, self).identify_as_bitbake()
+
     def supports(self, ud, d):
         """
         Check to see if a given url is for this fetcher
@@ -65,10 +80,11 @@ class Crate(Wget):
         # host (this is to allow custom crate registries to be specified
         host = '/'.join(parts[2:-2])
 
-        # if using upstream just fix it up nicely
+        # if using upstream fix it up nicely and make sure user-agent is set
         if host == 'crates.io':
             host = 'crates.io/api/v1/crates'
             cdn_host = 'static.crates.io/crates'
+            self.using_crates_io = True
         else:
             cdn_host = host
 
Signature · 1 line
2.34.1