fetch2: add Google Cloud Platform (GCP) fetcher
PATCH v2 1/1 fetch2: add Google Cloud Platform (GCP) fetcher
From: Emil Ekmečić <[email protected]>
This fetcher allows BitBake to fetch from a Google Cloud Storage bucket. The fetcher expects a gs:// URI of the following form:
SSTATE_MIRRORS = "file://.* gs://<bucket name>/PATH"
The fetcher uses the Google Cloud Storage Python Client, and expects it to be installed, configured, and authenticated prior to use.
Here is how this fetcher conforms to the fetcher expectations described at this link:
https://git.yoctoproject.org/poky/tree/bitbake/lib/bb/fetch2/README
a) Yes, network fetching only happens in the fetcher
b) The fetcher has nothing to do with the unpack phase so there is no network access there
c) This change doesn't affect the behavior of DL_DIR. The GCP fetcher only downloads to the DL_DIR in the same way that other fetchers, namely the S3 and Azure fetchers do.
d) The fetcher is identical to the S3 and Azure fetchers in this context
e) Yes, the fetcher output is deterministic because it is downloading tarballs from a bucket and not modifying them in any way.
f) I set up a local proxy using tinyproxy and set the http_proxy variable to test whether the Python API respected the proxy. It appears that it did as I could see traffic passing through the proxy. I also did some searching online and found posts indicating that the Google Cloud Python APIs supported the classic Linux proxy variables, namely:
- https://github.com/googleapis/google-api-python-client/issues/1260
g) Access is minimal, only checking if the file exists and downloading it if it does.
h) Not applicable, BitBake already knows which version it wants and the version infomation is encoded in the filename. The fetcher has no concept of versions.
i) Not applicable
j) Not applicable
k) No tests were added as part of this change. I didn't see any tests for the S3 or Azure changes either, is that OK?
l) I'm not 100% familiar but I don't believe this fetcher is using any tools during parse time. Please correct me if I'm wrong.
--- lib/bb/fetch2/__init__.py | 4 +- lib/bb/fetch2/gcp.py | 104 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 107 insertions(+), 1 deletion(-) create mode 100644 lib/bb/fetch2/gcp.py
lib/bb/fetch2/__init__.py +3 -1
@@ -1290,7 +1290,7 @@ class FetchData(object): if checksum_name in self.parm: checksum_expected = self.parm[checksum_name] - elif self.type not in ["http", "https", "ftp", "ftps", "sftp", "s3", "az", "crate"]: + elif self.type not in ["http", "https", "ftp", "ftps", "sftp", "s3", "az", "crate", "gs"]: checksum_expected = None else: checksum_expected = d.getVarFlag("SRC_URI", checksum_name)
@@ -1973,6 +1973,7 @@ from . import npm from . import npmsw from . import az from . import crate +from . import gcp methods.append(local.Local()) methods.append(wget.Wget())
@@ -1994,3 +1995,4 @@ methods.append(npm.Npm()) methods.append(npmsw.NpmShrinkWrap()) methods.append(az.Az()) methods.append(crate.Crate()) +methods.append(gcp.GCP())
lib/bb/fetch2/gcp.py new file +104 -0
104 lines not on this page · raw message
Signature · 1 line
2.40.1
Quoted 2 lines · eekmecic via lists.openembedded.org
e) Yes, the fetcher output is deterministic because it is downloading tarballs from a bucket and not modifying them in any way.
Let's say there's a supply chain attack on the bucket side, and the tarball has been replaced with a maliciously modified one. Will this fetcher be able to detect and reject such a download? For http tarball checksums are provided in the recipe. For git, the commit revision doubles as an integrity checksum (we trust that the local git client performs the check correctly). How is this handled here?
Alex
Quoted 6 lines · Alexander Kanavin
Quoted 2 lines · eekmecic via lists.openembedded.org
e) Yes, the fetcher output is deterministic because it is downloading tarballs from a bucket and not modifying them in any way.
Quoted 8 lines
Let's say there's a supply chain attack on the bucket side, and the tarball has been replaced with a maliciously modified one. Will this fetcher be able to detect and reject such a download? For http tarball checksums are provided in the recipe. For git, the commit revision doubles as an integrity checksum (we trust that the local git client performs the check correctly). How is this handled here?
Alex
This fetcher, in its current state, will not be able to detect a supply chain attack of the type you've described. It does no verification of any kind.
We are using this fetcher to support fetching sstate artifacts from a GCS bucket. From my understanding, these artifacts have signatures that are based on their inputs so BitBake would be able to verify that the expected signature matches the signature of the file downloaded by the fetcher.
I am unsure of how (if?) BitBake verifies the integrity of source tarballs downloaded from mirrors. We are not using it to fetch source, only prebuilt artifacts. I took a look at the other two fetchers that implement support for S3 and Azure and I don't see any differences in how they would handle a supply chain attack of the type you described. Please correct me if I'm wrong. Likewise please let me know if there's something that needs to be implemented in the fetcher code to prevent such an attack.
Emil
Quoted 8 lines · Emil Ekmečić
I am unsure of how (if?) BitBake verifies the integrity of source tarballs downloaded from mirrors. We are not using it to fetch source, only prebuilt artifacts. I took a look at the other two fetchers that implement support for S3 and Azure and I don't see any differences in how they would handle a supply chain attack of the type you described. Please correct me if I'm wrong. Likewise please let me know if there's something that needs to be implemented in the fetcher code to prevent such an attack.
Right, I thought you had actual recipes using the fetcher.
I think this bit:
def recommends_checksum(self, urldata):
return True
will require checksumming when the fetcher is used in a recipe. I don't actually know for sure what happens when the fetcher is used from sstate code and where is the code that prevents tampering with sstate objects.
Given that the fetcher is the same 'class' as s3/azure - seems primarily intended for sstate mirrors, I'm not sure if it should have tests in bitbake/lib/bb/tests/fetch.py. Maybe RP can clarify.
Otherwise, I'm fine with this.
Alex
Quoted 24 lines · Alexander Kanavin
Quoted 8 lines · Emil Ekmečić
I am unsure of how (if?) BitBake verifies the integrity of source tarballs downloaded from mirrors. We are not using it to fetch source, only prebuilt artifacts. I took a look at the other two fetchers that implement support for S3 and Azure and I don't see any differences in how they would handle a supply chain attack of the type you described. Please correct me if I'm wrong. Likewise please let me know if there's something that needs to be implemented in the fetcher code to prevent such an attack.
Right, I thought you had actual recipes using the fetcher.
I think this bit:
def
recommends_checksum(self, urldata):
return Truewill require checksumming when the fetcher is used in a recipe. I don't actually know for sure what happens when the fetcher is used from sstate code and where is the code that prevents tampering with sstate objects.
Quoted 7 lines
Given that the fetcher is the same 'class' as s3/azure - seems primarily intended for sstate mirrors, I'm not sure if it should have tests in
bitbake/lib/bb/tests/fetch.py. Maybe RP can clarify.Otherwise, I'm fine with this.
Alex
Yes, that bit will require checksumming. My colleague Étienne Cordonnier verified this by creating a recipe that uses this fetcher without specifying a checksum, which returned the following error:
WARNING: uutils-coreutils-0.0.16-r0 do_fetch: Missing checksum for '/workdir/downloads/sstate:wayland::1.20.0:r0::10:0004d8aa6640ecb43319a1c 2e68111d0522036c86fe0f331c1c830c0e0a30c31_populate_lic.tar.zst', consider adding at least one to the recipe:
SRC_URI[sha256sum] = "16c025cd29bc9eb535676160e0edaf90832b9b07204fe0f5e 180384b66c904ab"
After adding the checksum to the recipe as indicated by the warning, the warning message went away. Lastly, if he intentionally inputted an incorrect checksum BitBake threw an error:
ERROR: uutils-coreutils-0.0.16-r0 do_fetch: Fetcher failure for URL:
'gs://< our internal bucket name >/00/04/sstate:wayland::1.20.0:r0::1 0:0004d8aa6640ecb43319a1c2e68111d0522036c86fe0f331c1c830c0e0a30c3 1_populate_lic.tar.zst'. Checksum mismatch!
I will create another revision of the patch that includes documentation for the fetcher in the user manual. I will also create a patch in openembedded-core with some small changes needed for this feature to work. I'll ensure that I mention all the patches so that they can be integrated simultaneously with minimal breakage should they be accepted.
Thank you Alex for taking a look.
Emil