scripts/install-buildtools: add an option to specify where downloads go
PATCH v2 1/2 scripts/install-buildtools: add an option to specify where downloads go
From: Alexander Kanavin <[email protected]>
By default the script puts everything it downloads into a temporary directory and erases it after unpacking and installing the buildtools.
This isn't great for traceability and reproducibility of builds (being able to see what was downloaded exactly, and being able to reproduce setting up a build, especially if the buildtools download location isn't available for whatever reason).
This commit adds an option to download items into a specified directory and keep them there. I would particularly like to use it with bitbake-setup, where an optional feature to install the buildtools (exact implementation details tbd) would ensure the tarball remains available on local disk.
--- scripts/install-buildtools | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-)
scripts/install-buildtools +9 -4
@@ -142,6 +142,9 @@ def main(): default=DEFAULT_INSTALL_DIR, help='directory where buildtools SDK will be installed (optional)', action='store') + parser.add_argument('--downloads-directory', + help='use this directory for tarball/checksum downloads and do not erase them (default is a temporary directory which is deleted after unpacking and installing the buildtools)', + action='store') parser.add_argument('-r', '--release', default=DEFAULT_RELEASE, help='Yocto Project release string for SDK which will be '
@@ -235,11 +238,12 @@ def main(): safe_filename = quote(filename) buildtools_url = "%s/%s/buildtools/%s" % (base_url, args.release, safe_filename) - tmpsdk_dir = tempfile.mkdtemp() + sdk_dir = args.downloads_directory or tempfile.mkdtemp() + os.makedirs(sdk_dir, exist_ok=True) try: # Fetch installer logger.info("Fetching buildtools installer") - tmpbuildtools = os.path.join(tmpsdk_dir, filename) + tmpbuildtools = os.path.join(sdk_dir, filename) ret = subprocess.call("wget -q -O %s %s" % (tmpbuildtools, buildtools_url), shell=True) if ret != 0:
@@ -252,7 +256,7 @@ def main(): checksum_type = "sha256sum" check_url = "{}.{}".format(buildtools_url, checksum_type) checksum_filename = "{}.{}".format(filename, checksum_type) - tmpbuildtools_checksum = os.path.join(tmpsdk_dir, checksum_filename) + tmpbuildtools_checksum = os.path.join(sdk_dir, checksum_filename) ret = subprocess.call("wget -q -O %s %s" % (tmpbuildtools_checksum, check_url), shell=True) if ret != 0:
@@ -347,7 +351,8 @@ def main(): finally: # cleanup tmp directory - shutil.rmtree(tmpsdk_dir) + if not args.downloads_directory: + shutil.rmtree(sdk_dir) if __name__ == '__main__':
Signature · 1 line
2.39.5
PATCH v2 2/2 scripts/install-buildtools: write download URLs to files for traceability
From: Alexander Kanavin <[email protected]>
This extends the previous commit, so that download URLs are preserved in addition to actual artefacts. By default it's all written to a temporary directory and erased together with artefacts, but users can choose to preserve both:
$ install-buildtools -D --keep-downloads-directory=. ...
$ ls -1 buildtools_url checksum_url x86_64-buildtools-extended-nativesdk-standalone-5.1.2.sh x86_64-buildtools-extended-nativesdk-standalone-5.1.2.sh.sha256sum
Additionally, rename check_url variable into checksum_url as it's a better name.
--- scripts/install-buildtools | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-)
scripts/install-buildtools +7 -3
@@ -244,6 +244,8 @@ def main(): # Fetch installer logger.info("Fetching buildtools installer") tmpbuildtools = os.path.join(sdk_dir, filename) + with open(os.path.join(sdk_dir, 'buildtools_url'), 'w') as f: + f.write(buildtools_url) ret = subprocess.call("wget -q -O %s %s" % (tmpbuildtools, buildtools_url), shell=True) if ret != 0:
@@ -254,13 +256,15 @@ def main(): if args.check: logger.info("Fetching buildtools installer checksum") checksum_type = "sha256sum" - check_url = "{}.{}".format(buildtools_url, checksum_type) + checksum_url = "{}.{}".format(buildtools_url, checksum_type) checksum_filename = "{}.{}".format(filename, checksum_type) tmpbuildtools_checksum = os.path.join(sdk_dir, checksum_filename) + with open(os.path.join(sdk_dir, 'checksum_url'), 'w') as f: + f.write(checksum_url) ret = subprocess.call("wget -q -O %s %s" % - (tmpbuildtools_checksum, check_url), shell=True) + (tmpbuildtools_checksum, checksum_url), shell=True) if ret != 0: - logger.error("Could not download file from %s" % check_url) + logger.error("Could not download file from %s" % checksum_url) return ret regex = re.compile(r"^(?P<checksum>[0-9a-f]+)\s+(?P<path>.*/)?(?P<filename>.*)$") with open(tmpbuildtools_checksum, 'rb') as f:
Signature · 1 line
2.39.5
Patchtest results for [PATCH v2 1/2] scripts/install-buildtools: add an option to specify where downloads go
In reply to Alexander KanavinThank you for your submission. Patchtest identified one or more issues with the patch. Please see the log below for more information:
---
Testing patch /home/patchtest/share/mboxes/v2-1-2-scripts-install-buildtools-add-an-option-to-specify-where-downloads-go.patch
FAIL: test max line length: Patch line too long (current length 201, maximum is 200) (test_metadata.TestMetadata.test_max_line_length)
PASS: test Signed-off-by presence (test_mbox.TestMbox.test_signed_off_by_presence)
PASS: test author valid (test_mbox.TestMbox.test_author_valid)
PASS: test commit message presence (test_mbox.TestMbox.test_commit_message_presence)
PASS: test commit message user tags (test_mbox.TestMbox.test_commit_message_user_tags)
PASS: test mbox format (test_mbox.TestMbox.test_mbox_format)
PASS: test non-AUH upgrade (test_mbox.TestMbox.test_non_auh_upgrade)
PASS: test shortlog format (test_mbox.TestMbox.test_shortlog_format)
PASS: test shortlog length (test_mbox.TestMbox.test_shortlog_length)
PASS: test target mailing list (test_mbox.TestMbox.test_target_mailing_list)
SKIP: pretest pylint: No python related patches, skipping test (test_python_pylint.PyLint.pretest_pylint)
SKIP: pretest src uri left files: No modified recipes, skipping pretest (test_metadata.TestMetadata.pretest_src_uri_left_files)
SKIP: test CVE check ignore: No modified recipes or older target branch, skipping test (test_metadata.TestMetadata.test_cve_check_ignore)
SKIP: test CVE tag format: No new CVE patches introduced (test_patch.TestPatch.test_cve_tag_format)
SKIP: test Signed-off-by presence: No new CVE patches introduced (test_patch.TestPatch.test_signed_off_by_presence)
SKIP: test Upstream-Status presence: No new CVE patches introduced (test_patch.TestPatch.test_upstream_status_presence_format)
SKIP: test bugzilla entry format: No bug ID found (test_mbox.TestMbox.test_bugzilla_entry_format)
SKIP: test lic files chksum modified not mentioned: No modified recipes, skipping test (test_metadata.TestMetadata.test_lic_files_chksum_modified_not_mentioned)
SKIP: test lic files chksum presence: No added recipes, skipping test (test_metadata.TestMetadata.test_lic_files_chksum_presence)
SKIP: test license presence: No added recipes, skipping test (test_metadata.TestMetadata.test_license_presence)
SKIP: test pylint: No python related patches, skipping test (test_python_pylint.PyLint.test_pylint)
SKIP: test series merge on head: Merge test is disabled for now (test_mbox.TestMbox.test_series_merge_on_head)
SKIP: test src uri left files: No modified recipes, skipping pretest (test_metadata.TestMetadata.test_src_uri_left_files)
SKIP: test summary presence: No added recipes, skipping test (test_metadata.TestMetadata.test_summary_presence)
---
Please address the issues identified and
submit a new revision of the patch, or alternatively, reply to this email with an explanation of why the patch should be accepted. If you believe these results are due to an error in patchtest, please submit a bug at https://bugzilla.yoctoproject.org/ (use the 'Patchtest' category under 'Yocto Project Subprojects'). For more information on specific failures, see: https://wiki.yoctoproject.org/wiki/Patchtest. Thank you!
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [OE-core] [PATCH v2 2/2] scripts/install-buildtools: write download URLs to files for traceability
Quoted 1 line · Alexander Kanavin via lists.openembedded.org
$ install-buildtools -D --keep-downloads-directory=.
--downloads-directory as of v2 :) I forgot to tweak this commit message.
Alex