Add utils/source-check
cover Add utils/source-check
From: Thomas De Schampheleire <[email protected]>
Hello,
Buildroot used to have a 'make source-check' command, which was later removed. Nevertheless, for organizations that use a download mirror and want to efficiently check that all needed files are present without actually downloading, an equivalent is still useful.
After the addition of the 'show-info' target, this is now possible without polluting the core Buildroot infrastructure.
The script is a 'utility', it is not used automatically, just present for the benefit of users.
Note: as mentioned in the commit msg, the current script only covers scp and hg downloads, my needs. It could be extended to cover git etc., but it may not be possible to check remotely if a git revision exists on a repository. So for these additional download methods, it may be needed to perform an actual clone?
Patch 1 adds BR2_PRIMARY_SITE_ONLY_EXTENDED_DOMAINS and is useful independently from source-check. It addresses the problem where BR2_PRIMARY_SITE_ONLY cannot be used if you also have packages under version control.
Patch 2 adds the source-check script.
Best regards,
Thomas
Thomas De Schampheleire (2):
core: add BR2_PRIMARY_SITE_ONLY_EXTENDED_DOMAINS utils/source-check: new script
Config.in | 11 +++ package/pkg-download.mk | 8 +- utils/source-check | 170 ++++++++++++++++++++++++++++++++++++++++ 3 files changed, 188 insertions(+), 1 deletion(-) create mode 100755 utils/source-check
Signature · 1 line
2.26.2
PATCH 1/2 core: add BR2_PRIMARY_SITE_ONLY_EXTENDED_DOMAINS
From: Thomas De Schampheleire <[email protected]>
If configured, the primary site typically points to a mirror on the intranet of an organization. The purpose of BR2_PRIMARY_SITE_ONLY is then to only download from this mirror.
However, the organization may also have some local Buildroot packages that download from a version control repository (git, hg, ...). In this case, the mirror will normally not contain the sources, instead they should be cloned via the version control tool. So in this case, BR2_PRIMARY_SITE_ONLY cannot be used.
This means that the organization must resort to other means to make sure no external downloads are performed.
This patch attempts to solve this situation by adding BR2_PRIMARY_SITE_ONLY_EXTENDED_DOMAINS. This string option can contain additional domains from which download is allowed when BR2_PRIMARY_SITE_ONLY is set.
The organization can thus set:
BR2_PRIMARY_SITE_ONLY=y BR2_PRIMARY_SITE_ONLY_EXTENDED_DOMAINS="git.example.com hg.example.com"
to disallow any external downloads other than the primary site and the mentioned version control domains.
--- Config.in | 11 +++++++++++ package/pkg-download.mk | 8 +++++++- 2 files changed, 18 insertions(+), 1 deletion(-)
Config.in +11 -0
@@ -226,6 +226,17 @@ config BR2_PRIMARY_SITE_ONLY the project can be built even if the upstream tarball locations disappear. +config BR2_PRIMARY_SITE_ONLY_EXTENDED_DOMAINS + string "Additional domains to allow downloads from" + depends on BR2_PRIMARY_SITE_ONLY + help + If BR2_PRIMARY_SITE_ONLY is enabled, version control downloads + (git, hg, ...) on the 'internal' domain would also be disallowed. + With this option, you can specify additional domains from + which downloads will be allowed in BR2_PRIMARY_SITE_ONLY-mode. + Domains should not include a protocol prefix, and multiple + domains can be separated by spaces. + if !BR2_PRIMARY_SITE_ONLY config BR2_BACKUP_SITE
package/pkg-download.mk +7 -1
@@ -78,7 +78,13 @@ DOWNLOAD_URIS += \ $(call getschemeplusuri,$(call qstrip,$(BR2_PRIMARY_SITE)),urlencode) endif -ifeq ($(BR2_PRIMARY_SITE_ONLY),) +ifeq ($(BR2_PRIMARY_SITE_ONLY),y) +# Conditionally add site download if it matches the configured extended domains +DOWNLOAD_URIS += \ + $(if $(filter $(call qstrip,$(BR2_PRIMARY_SITE_ONLY_EXTENDED_DOMAINS)),$(call domain,$(1))), \ + $(patsubst %/,%,$(dir $(call qstrip,$(1))))) +else +# Unconditionally add site download DOWNLOAD_URIS += \ $(patsubst %/,%,$(dir $(call qstrip,$(1)))) ifneq ($(call qstrip,$(BR2_BACKUP_SITE)),)
Signature · 1 line
2.26.2
PATCH 2/2 utils/source-check: new script
In reply to Thomas De SchampheleireFrom: Thomas De Schampheleire <[email protected]>
This source-check script is a replacement for 'make source-check' that existed earlier.
It takes as input a list of defconfigs, and then efficiently determines whether all files needed can be downloaded.
The settings of BR2_PRIMARY_SITE, BR2_PRIMARY_SITE_ONLY and BR2_PRIMARY_SITE_ONLY_EXTENDED_DOMAINS will be used as specified in the respective defconfigs.
Current limitations:
- only covers scp and hg URIs, not git or others.
--- utils/source-check | 170 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 170 insertions(+) create mode 100755 utils/source-check
utils/source-check new file +170 -0
@@ -0,0 +1,170 @@ +#!/usr/bin/env python3 + +import json +import multiprocessing +import os +import shutil +import subprocess +import sys + +# example input: +# 'rsync': {'dependencies': ['host-ccache', +# 'host-skeleton', +# 'host-tar', +# 'popt', +# 'skeleton', +# 'toolchain', +# 'zlib'], +# 'dl_dir': 'rsync', +# 'downloads': [{'source': 'rsync-3.1.3.tar.gz', +# 'uris': ['scp|urlencode+scp://xxx at mirror.example.com/rsync', +# 'scp|urlencode+scp://xxx at mirror.example.com', +# 'http+http://rsync.samba.org/ftp/rsync/src', +# 'http|urlencode+http://sources.buildroot.net/rsync', +# 'http|urlencode+http://sources.buildroot.net']}], +# 'install_images': False, +# 'install_staging': False, +# 'install_target': True, +# 'licenses': 'GPL-3.0+', +# 'reverse_dependencies': [], +# 'type': 'target', +# 'version': '3.1.3', +# 'virtual': False}, +# + +def get_files_to_check_one_defconfig(defconfig): + outputdir = 'sourcecheck_%s' % defconfig + subprocess.check_call(['make', '--no-print-directory', '-s', defconfig, 'O=%s' % outputdir]) + # Note: set suitable-host-package to empty to pretend no suitable tools are + # present on the host, and thus force all potentially-needed sources in the + # list (e.g. cmake, gzip, ...) + output = subprocess.check_output(['make', '--no-print-directory', '-s', 'show-info', 'O=%s' % outputdir, 'suitable-host-package=']) + info = json.loads(output) + + files_to_check = set() + + for pkg in info: + if not 'downloads' in info[pkg]: + sys.stderr.write("Warning: %s: no downloads for package '%s'\n" % (defconfig, pkg)) + continue + if not info[pkg]['downloads']: + sys.stderr.write("Warning: %s: empty downloads for package '%s'\n" % (defconfig, pkg)) + continue + for download in info[pkg]['downloads']: + if not 'source' in download: + sys.stderr.write("Warning: %s: no source filename found for package '%s'\n" % (defconfig, pkg)) + continue + if not 'uris' in download: + sys.stderr.write("Warning: %s: no uri's found for package '%s'\n" % (defconfig, pkg)) + continue + + # tuple: (pkg, version, filename, uris) + # Note: host packages have the same sources as for target, so strip + # the 'host-' prefix. Because we are using a set, this will remove + # duplicate entries. + pkgname = pkg[5:] if pkg.startswith('host-') else pkg + files_to_check.add(( + pkgname, + info[pkg]['version'], + download['source'], + tuple([uri for uri in download['uris']]), + )) + + shutil.rmtree(outputdir) + return files_to_check + +def get_files_to_check(defconfigs): + total_files_to_check = set() + + num_processes = multiprocessing.cpu_count() * 2 + print('Dispatching over %s processes' % num_processes) + with multiprocessing.Pool(processes=num_processes) as pool: + result_objs = [ + pool.apply_async(get_files_to_check_one_defconfig, (defconfig,)) + for defconfig in defconfigs + ] + results = [p.get() for p in result_objs] + + for result in results: + total_files_to_check |= result + + return total_files_to_check + +def sourcecheck_one_uri(pkg, version, filename, uri): + + def sourcecheck_scp(pkg, version, filename, uri): + real_uri = uri.split('+', 1)[1] + '/' + filename + if real_uri.startswith('scp://'): + real_uri = real_uri[6:] + domain, path = real_uri.split(':', 1) + with open(os.devnull, 'w') as devnull: + ret = subprocess.call(['ssh', domain, 'test', '-f', path], stderr=devnull) + return ret == 0 + + def sourcecheck_hg(pkg, version, filename, uri): + real_uri = uri.split('+', 1)[1] + with open(os.devnull, 'w') as devnull: + ret = subprocess.call(['hg', 'identify', '--rev', version, real_uri], stdout=devnull, stderr=devnull) + return ret == 0 + + if uri.startswith('scp'): + handler = sourcecheck_scp + elif uri.startswith('hg'): + handler = sourcecheck_hg + else: + raise Exception("Cannot handle unknown URI type: '%s' for package '%s'" % (uri, pkg)) + + return handler(pkg, version, filename, uri) + +def sourcecheck_one_file(pkg, version, filename, uris): + result = any(sourcecheck_one_uri(pkg, version, filename, uri) for uri in uris) + return pkg, version, filename, result + +def sourcecheck(files_to_check): + + def process_result(result): + pkg, version, filename, success = result + if success: + print(' OK: pkg %s, filename %s' % (pkg, filename)) + else: + sys.stderr.write('NOK: pkg %s, filename %s -- ERROR!\n' % (pkg, filename)) + + num_processes = multiprocessing.cpu_count() * 2 + print('Dispatching over %s processes' % num_processes) + with multiprocessing.Pool(processes=num_processes) as pool: + result_objs = [ + pool.apply_async(sourcecheck_one_file, entry, callback=process_result) + for entry in files_to_check + ] + results = [p.get() for p in result_objs] + + succeeded = [(pkg, version, filename, success) for (pkg, version, filename, success) in results if success] + failed = [(pkg, version, filename, success) for (pkg, version, filename, success) in results if not success] + + print('\nSummary: %s OK, %s NOK, %s total' % (len(succeeded), len(failed), len(results))) + + if len(failed): + print('\nFAILED FILES') + for pkg, version, filename, success in sorted(failed): + print('pkg: %s, version: %s, file: %s/%s' % (pkg, version, pkg, filename)) + + return len(failed) == 0 + +def main(): + # - run 'make show-info' for each defconfig + # - determine the unique urls to check (take into account that one package + # can have multiple urls) + # - check each unique url + + defconfigs = sys.argv[1:] + if not defconfigs: + sys.stderr.write('Error: pass list of defconfigs as arguments\n') + sys.exit(1) + + total_files_to_check = get_files_to_check(defconfigs) + return sourcecheck(total_files_to_check) + +if __name__ == '__main__': + ret = main() + if not ret: + sys.exit(1)
Signature · 1 line
2.26.2
cover Add utils/source-check
In reply to Thomas De SchampheleireEl lun., 4 may. 2020 a las 13:14, Thomas De Schampheleire (< [email protected]>) escribi?:
Quoted 43 lines
From: Thomas De Schampheleire <[email protected]>
Hello,
Buildroot used to have a 'make source-check' command, which was later removed.
Nevertheless, for organizations that use a download mirror and want to efficiently check that all needed files are present without actually downloading, an equivalent is still useful.After the addition of the 'show-info' target, this is now possible without polluting the core Buildroot infrastructure.
The script is a 'utility', it is not used automatically, just present for the
benefit of users.Note: as mentioned in the commit msg, the current script only covers scp and hg
downloads, my needs. It could be extended to cover git etc., but it may not be
possible to check remotely if a git revision exists on a repository. So for these additional download methods, it may be needed to perform an actual clone?Patch 1 adds
BR2_PRIMARY_SITE_ONLY_EXTENDED_DOMAINSand is useful independently
from source-check. It addresses the problem whereBR2_PRIMARY_SITE_ONLYcannot
be used if you also have packages under version control.Patch 2 adds the source-check script.
Best regards,
ThomasThomas De Schampheleire (2):
core: addBR2_PRIMARY_SITE_ONLY_EXTENDED_DOMAINSutils/source-check: new script
Quoted 10 lines
Config.in | 11 +++ package/pkg-download.mk | 8 +- utils/source-check | 170 ++++++++++++++++++++++++++++++++++++++++ 3 files changed, 188 insertions(+), 1 deletion(-) create mode 100755 utils/source-check--
2.26.2
I would like to ping this series for feedback:
http://patchwork.ozlabs.org/project/buildroot/patch/20200504111551.15920-2-patrickdepinguin at gmail.com/
http://patchwork.ozlabs.org/project/buildroot/patch/20200504111551.15920-3-patrickdepinguin at gmail.com/ -------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20200804/04c9525c/attachment.html>