gcp.py: remove slow calls to gsutil stat
PATCH 1/1 gcp.py: remove slow calls to gsutil stat
From: Etienne Cordonnier <[email protected]>
The changes of 1ab1d36c0af6fc58a974106b61ff4d37da6cb229 added calls to "gsutil stat" to avoid unhandled exceptions, however:
- in the case of checkstatus() this is redundant with the call to self.gcp_client.bucket(ud.host).blob(path).exists() which already returns True/False
and does not throw an exception in case the file does not exist.
- Also the call to gsutil stat is much slower than using the python client to call exists() so we should not replace the call to exists() with a call to gsutil stat.
- I think the intent of calling check_network_access in checkstatus() was to error-out in case the error is disabled. We can rather change the string "gsutil stat" to something else to make the code more readable.
- add a try/except block in download() instead of the extra call to gsutil
--- lib/bb/fetch2/gcp.py | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-)
lib/bb/fetch2/gcp.py +7 -7
@@ -19,11 +19,11 @@ Additionally, gsutil must also be installed. import os import bb +from google.api_core.exceptions import NotFound import urllib.parse, urllib.error from bb.fetch2 import FetchMethod from bb.fetch2 import FetchError from bb.fetch2 import logger -from bb.fetch2 import runfetchcmd class GCP(FetchMethod): """
@@ -48,7 +48,6 @@ class GCP(FetchMethod): ud.basename = os.path.basename(ud.path) ud.localfile = d.expand(urllib.parse.unquote(ud.basename)) - ud.basecmd = "gsutil stat" def get_gcp_client(self): from google.cloud import storage
@@ -63,13 +62,15 @@ class GCP(FetchMethod): if self.gcp_client is None: self.get_gcp_client() - bb.fetch2.check_network_access(d, ud.basecmd, f"gs://{ud.host}{ud.path}") - runfetchcmd("%s %s" % (ud.basecmd, f"gs://{ud.host}{ud.path}"), d) + bb.fetch2.check_network_access(d, "blob.download_to_filename", f"gs://{ud.host}{ud.path}") # Path sometimes has leading slash, so strip it path = ud.path.lstrip("/") blob = self.gcp_client.bucket(ud.host).blob(path) - blob.download_to_filename(ud.localpath) + try: + blob.download_to_filename(ud.localpath) + except NotFound: + raise FetchError("The GCP API threw a NotFound exception") # Additional sanity checks copied from the wget class (although there # are no known issues which mean these are required, treat the GCP API
@@ -91,8 +92,7 @@ class GCP(FetchMethod): if self.gcp_client is None: self.get_gcp_client() - bb.fetch2.check_network_access(d, ud.basecmd, f"gs://{ud.host}{ud.path}") - runfetchcmd("%s %s" % (ud.basecmd, f"gs://{ud.host}{ud.path}"), d) + bb.fetch2.check_network_access(d, "gcp_client.bucket(ud.host).blob(path).exists()", f"gs://{ud.host}{ud.path}") # Path sometimes has leading slash, so strip it path = ud.path.lstrip("/")
Signature · 1 line
2.43.0
Quoted 23 lines · Etienne Cordonnier via lists.openembedded.org
From: Etienne Cordonnier <[email protected]>
The changes of
1ab1d36c0af6fc58a974106b61ff4d37da6cb229added calls to "gsutil stat" to avoid unhandled exceptions, however:
- in the case ofcheckstatus()this is redundant with the call to self.gcp_client.bucket(ud.host).blob(path).exists() which already returns True/False
and does not throw an exception in case the file does not exist.
- Also the call to gsutil stat is much slower than using the python client to callexists()so we should not replace the call toexists()with a call to gsutil stat.
- I think the intent of callingcheck_network_accessincheckstatus()was to error-out in case the error is disabled. We can rather change the string "gsutil stat" to something else to make the code more readable.
- add a try/except block indownload()instead of the extra call to gsutilSigned-off-by Etienne Cordonnier <[email protected]>---
lib/bb/fetch2/gcp.py | 14 +++++++-------1 file changed, 7 insertions(+), 7 deletions(-)lib/bb/fetch2/gcp.py +0
-0@@ -19,11 +19,11 @@ Additionally, gsutil must also be installed.
import os
import bb
+from google.api_core.exceptions import NotFound
Quoted 8 lines
import urllib.parse, urllib.error
from bb.fetch2 import FetchMethod
from bb.fetch2 import FetchError
from bb.fetch2 import logger
-from bb.fetch2 import runfetchcmd
class GCP(FetchMethod):
"""
I put this into a testing branch and everything broke since the google module isn't present in most places. I've moved the import into the download method.
Cheers,
Richard
Makes sense, I had forgotten that the module can be included inside functions.
Étienne
Quoted 50 lines · Richard Purdie
Quoted 3 lines · Etienne Cordonnier via lists.openembedded.org
From: Etienne Cordonnier <[email protected]>
The changes of
1ab1d36c0af6fc58a974106b61ff4d37da6cb229added calls to"gsutil stat" to avoid unhandled exceptions, however:
Quoted 1 line
- in the case of
checkstatus()this is redundant with the call toself.gcp_client.bucket(ud.host).blob(path).exists() which already returns True/False
Quoted 2 lines
and does not throw an exception in case the file does not exist.
- Also the call to gsutil stat is much slower than using the pythonclient to call
exists()so we should not replace the call toexists()with a call to gsutil stat.Quoted 1 line
- I think the intent of calling
check_network_accessincheckstatus()was to error-out in case the error is disabled. We can rather change the string "gsutil stat" to something else to make the code more readable.
Quoted 1 line
- add a try/except block in
download()instead of the extra call togsutil
Quoted 13 lines
Signed-off-by Etienne Cordonnier <[email protected]>--- lib/bb/fetch2/gcp.py | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-)lib/bb/fetch2/gcp.py +0
-0@@ -19,11 +19,11 @@ Additionally, gsutil must also be installed. import oslib/bb/fetch2/gcp.py Additionally, gsutil must also be installed. · line 29
Quoted 10 lines
lib/bb/fetch2/gcp.py +1
-1import bb +from google.api_core.exceptions import NotFound import urllib.parse, urllib.error from bb.fetch2 import FetchMethod from bb.fetch2 import FetchError from bb.fetch2 import logger -from bb.fetch2 import runfetchcmd class GCP(FetchMethod): """I put this into a testing branch and everything broke since the google module isn't present in most places. I've moved the import into the download method.
Cheers,
Richard