Skip to contentSkip to conversation

Add utils/source-check

PATCH New Thomas De Schampheleire 1 participant 4 messages May 4, 2020 – Aug 4, 2020

lore
Share & tools
lore
#0 lore raw

cover Add utils/source-check

From: Thomas De Schampheleire <[email protected]>

Hello,

Buildroot used to have a 'make source-check' command, which was later removed. Nevertheless, for organizations that use a download mirror and want to efficiently check that all needed files are present without actually downloading, an equivalent is still useful.

After the addition of the 'show-info' target, this is now possible without polluting the core Buildroot infrastructure.

The script is a 'utility', it is not used automatically, just present for the benefit of users.

Note: as mentioned in the commit msg, the current script only covers scp and hg downloads, my needs. It could be extended to cover git etc., but it may not be possible to check remotely if a git revision exists on a repository. So for these additional download methods, it may be needed to perform an actual clone?

Patch 1 adds BR2_PRIMARY_SITE_ONLY_EXTENDED_DOMAINS and is useful independently from source-check. It addresses the problem where BR2_PRIMARY_SITE_ONLY cannot be used if you also have packages under version control.

Patch 2 adds the source-check script.

Best regards,
Thomas

Thomas De Schampheleire (2):
core: add BR2_PRIMARY_SITE_ONLY_EXTENDED_DOMAINS utils/source-check: new script

 Config.in               |  11 +++
 package/pkg-download.mk |   8 +-
 utils/source-check      | 170 ++++++++++++++++++++++++++++++++++++++++
 3 files changed, 188 insertions(+), 1 deletion(-)
 create mode 100755 utils/source-check
Signature · 1 line
2.26.2
#1 lore raw

PATCH 1/2 core: add BR2_PRIMARY_SITE_ONLY_EXTENDED_DOMAINS

From: Thomas De Schampheleire <[email protected]>

If configured, the primary site typically points to a mirror on the intranet of an organization. The purpose of BR2_PRIMARY_SITE_ONLY is then to only download from this mirror.

However, the organization may also have some local Buildroot packages that download from a version control repository (git, hg, ...). In this case, the mirror will normally not contain the sources, instead they should be cloned via the version control tool. So in this case, BR2_PRIMARY_SITE_ONLY cannot be used.

This means that the organization must resort to other means to make sure no external downloads are performed.

This patch attempts to solve this situation by adding BR2_PRIMARY_SITE_ONLY_EXTENDED_DOMAINS. This string option can contain additional domains from which download is allowed when BR2_PRIMARY_SITE_ONLY is set.

The organization can thus set:
BR2_PRIMARY_SITE_ONLY=y BR2_PRIMARY_SITE_ONLY_EXTENDED_DOMAINS="git.example.com hg.example.com"

to disallow any external downloads other than the primary site and the mentioned version control domains.

---
 Config.in               | 11 +++++++++++
 package/pkg-download.mk |  8 +++++++-
 2 files changed, 18 insertions(+), 1 deletion(-)
Config.in +11 -0
index 48fd36ff98..0843215bc5 100644
@@ -226,6 +226,17 @@ config BR2_PRIMARY_SITE_ONLY
 	  the project can be built even if the upstream tarball
 	  locations disappear.
 
+config BR2_PRIMARY_SITE_ONLY_EXTENDED_DOMAINS
+	string "Additional domains to allow downloads from"
+	depends on BR2_PRIMARY_SITE_ONLY
+	help
+	  If BR2_PRIMARY_SITE_ONLY is enabled, version control downloads
+	  (git, hg, ...) on the 'internal' domain would also be disallowed.
+	  With this option, you can specify additional domains from
+	  which downloads will be allowed in BR2_PRIMARY_SITE_ONLY-mode.
+	  Domains should not include a protocol prefix, and multiple
+	  domains can be separated by spaces.
+
 if !BR2_PRIMARY_SITE_ONLY
 
 config BR2_BACKUP_SITE
package/pkg-download.mk +7 -1
index de619ba90a..f189dba233 100644
@@ -78,7 +78,13 @@ DOWNLOAD_URIS += \
 	$(call getschemeplusuri,$(call qstrip,$(BR2_PRIMARY_SITE)),urlencode)
 endif
 
-ifeq ($(BR2_PRIMARY_SITE_ONLY),)
+ifeq ($(BR2_PRIMARY_SITE_ONLY),y)
+# Conditionally add site download if it matches the configured extended domains
+DOWNLOAD_URIS += \
+	$(if $(filter $(call qstrip,$(BR2_PRIMARY_SITE_ONLY_EXTENDED_DOMAINS)),$(call domain,$(1))), \
+		$(patsubst %/,%,$(dir $(call qstrip,$(1)))))
+else
+# Unconditionally add site download
 DOWNLOAD_URIS += \
 	$(patsubst %/,%,$(dir $(call qstrip,$(1))))
 ifneq ($(call qstrip,$(BR2_BACKUP_SITE)),)
Signature · 1 line
2.26.2
#2 lore raw

PATCH 2/2 utils/source-check: new script

In reply to Thomas De Schampheleire

From: Thomas De Schampheleire <[email protected]>

This source-check script is a replacement for 'make source-check' that existed earlier.

It takes as input a list of defconfigs, and then efficiently determines whether all files needed can be downloaded.

The settings of BR2_PRIMARY_SITE, BR2_PRIMARY_SITE_ONLY and BR2_PRIMARY_SITE_ONLY_EXTENDED_DOMAINS will be used as specified in the respective defconfigs.

Current limitations:
- only covers scp and hg URIs, not git or others.

---
 utils/source-check | 170 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 170 insertions(+)
 create mode 100755 utils/source-check
utils/source-check new file +170 -0
new file mode 100755
index 0000000000..a387f71d33
@@ -0,0 +1,170 @@
+#!/usr/bin/env python3
+
+import json
+import multiprocessing
+import os
+import shutil
+import subprocess
+import sys
+
+# example input:
+# 'rsync': {'dependencies': ['host-ccache',
+#                            'host-skeleton',
+#                            'host-tar',
+#                            'popt',
+#                            'skeleton',
+#                            'toolchain',
+#                            'zlib'],
+#           'dl_dir': 'rsync',
+#           'downloads': [{'source': 'rsync-3.1.3.tar.gz',
+#                          'uris': ['scp|urlencode+scp://xxx at mirror.example.com/rsync',
+#                                   'scp|urlencode+scp://xxx at mirror.example.com',
+#                                   'http+http://rsync.samba.org/ftp/rsync/src',
+#                                   'http|urlencode+http://sources.buildroot.net/rsync',
+#                                   'http|urlencode+http://sources.buildroot.net']}],
+#           'install_images': False,
+#           'install_staging': False,
+#           'install_target': True,
+#           'licenses': 'GPL-3.0+',
+#           'reverse_dependencies': [],
+#           'type': 'target',
+#           'version': '3.1.3',
+#           'virtual': False},
+#
+
+def get_files_to_check_one_defconfig(defconfig):
+    outputdir = 'sourcecheck_%s' % defconfig
+    subprocess.check_call(['make', '--no-print-directory', '-s', defconfig, 'O=%s' % outputdir])
+    # Note: set suitable-host-package to empty to pretend no suitable tools are
+    # present on the host, and thus force all potentially-needed sources in the
+    # list (e.g. cmake, gzip, ...)
+    output = subprocess.check_output(['make', '--no-print-directory', '-s', 'show-info', 'O=%s' % outputdir, 'suitable-host-package='])
+    info = json.loads(output)
+
+    files_to_check = set()
+
+    for pkg in info:
+        if not 'downloads' in info[pkg]:
+            sys.stderr.write("Warning: %s: no downloads for package '%s'\n" % (defconfig, pkg))
+            continue
+        if not info[pkg]['downloads']:
+            sys.stderr.write("Warning: %s: empty downloads for package '%s'\n" % (defconfig, pkg))
+            continue
+        for download in info[pkg]['downloads']:
+            if not 'source' in download:
+                sys.stderr.write("Warning: %s: no source filename found for package '%s'\n" % (defconfig, pkg))
+                continue
+            if not 'uris' in download:
+                sys.stderr.write("Warning: %s: no uri's found for package '%s'\n" % (defconfig, pkg))
+                continue
+
+            # tuple: (pkg, version, filename, uris)
+            # Note: host packages have the same sources as for target, so strip
+            # the 'host-' prefix. Because we are using a set, this will remove
+            # duplicate entries.
+            pkgname = pkg[5:] if pkg.startswith('host-') else pkg
+            files_to_check.add((
+                pkgname,
+                info[pkg]['version'],
+                download['source'],
+                tuple([uri for uri in download['uris']]),
+            ))
+
+    shutil.rmtree(outputdir)
+    return files_to_check
+
+def get_files_to_check(defconfigs):
+    total_files_to_check = set()
+
+    num_processes = multiprocessing.cpu_count() * 2
+    print('Dispatching over %s processes' % num_processes)
+    with multiprocessing.Pool(processes=num_processes) as pool:
+        result_objs = [
+            pool.apply_async(get_files_to_check_one_defconfig, (defconfig,))
+            for defconfig in defconfigs
+        ]
+        results = [p.get() for p in result_objs]
+
+    for result in results:
+        total_files_to_check |= result
+
+    return total_files_to_check
+
+def sourcecheck_one_uri(pkg, version, filename, uri):
+
+    def sourcecheck_scp(pkg, version, filename, uri):
+        real_uri = uri.split('+', 1)[1] + '/' + filename
+        if real_uri.startswith('scp://'):
+            real_uri = real_uri[6:]
+        domain, path = real_uri.split(':', 1)
+        with open(os.devnull, 'w') as devnull:
+            ret = subprocess.call(['ssh', domain, 'test', '-f', path], stderr=devnull)
+        return ret == 0
+
+    def sourcecheck_hg(pkg, version, filename, uri):
+        real_uri = uri.split('+', 1)[1]
+        with open(os.devnull, 'w') as devnull:
+            ret = subprocess.call(['hg', 'identify', '--rev', version, real_uri], stdout=devnull, stderr=devnull)
+        return ret == 0
+
+    if uri.startswith('scp'):
+        handler = sourcecheck_scp
+    elif uri.startswith('hg'):
+        handler = sourcecheck_hg
+    else:
+        raise Exception("Cannot handle unknown URI type: '%s' for package '%s'" % (uri, pkg))
+
+    return handler(pkg, version, filename, uri)
+
+def sourcecheck_one_file(pkg, version, filename, uris):
+    result = any(sourcecheck_one_uri(pkg, version, filename, uri) for uri in uris)
+    return pkg, version, filename, result
+
+def sourcecheck(files_to_check):
+
+    def process_result(result):
+        pkg, version, filename, success = result
+        if success:
+            print(' OK: pkg %s, filename %s' % (pkg, filename))
+        else:
+            sys.stderr.write('NOK: pkg %s, filename %s -- ERROR!\n' % (pkg, filename))
+
+    num_processes = multiprocessing.cpu_count() * 2
+    print('Dispatching over %s processes' % num_processes)
+    with multiprocessing.Pool(processes=num_processes) as pool:
+        result_objs = [
+            pool.apply_async(sourcecheck_one_file, entry, callback=process_result)
+            for entry in files_to_check
+        ]
+        results = [p.get() for p in result_objs]
+
+    succeeded = [(pkg, version, filename, success) for (pkg, version, filename, success) in results if success]
+    failed = [(pkg, version, filename, success) for (pkg, version, filename, success) in results if not success]
+
+    print('\nSummary: %s OK, %s NOK, %s total' % (len(succeeded), len(failed), len(results)))
+
+    if len(failed):
+        print('\nFAILED FILES')
+        for pkg, version, filename, success in sorted(failed):
+            print('pkg: %s, version: %s, file: %s/%s' % (pkg, version, pkg, filename))
+
+    return len(failed) == 0
+
+def main():
+    # - run 'make show-info' for each defconfig
+    # - determine the unique urls to check (take into account that one package
+    #       can have multiple urls)
+    # - check each unique url
+
+    defconfigs = sys.argv[1:]
+    if not defconfigs:
+        sys.stderr.write('Error: pass list of defconfigs as arguments\n')
+        sys.exit(1)
+
+    total_files_to_check = get_files_to_check(defconfigs)
+    return sourcecheck(total_files_to_check)
+
+if __name__ == '__main__':
+    ret = main()
+    if not ret:
+        sys.exit(1)
Signature · 1 line
2.26.2
#3 lore raw

cover Add utils/source-check

In reply to Thomas De Schampheleire

El lun., 4 may. 2020 a las 13:14, Thomas De Schampheleire (< [email protected]>) escribi?:

Quoted 43 lines

From: Thomas De Schampheleire <[email protected]>

Hello,

Buildroot used to have a 'make source-check' command, which was later removed.
Nevertheless, for organizations that use a download mirror and want to efficiently check that all needed files are present without actually downloading, an equivalent is still useful.

After the addition of the 'show-info' target, this is now possible without polluting the core Buildroot infrastructure.

The script is a 'utility', it is not used automatically, just present for the
benefit of users.

Note: as mentioned in the commit msg, the current script only covers scp and hg
downloads, my needs. It could be extended to cover git etc., but it may not be
possible to check remotely if a git revision exists on a repository. So for these additional download methods, it may be needed to perform an actual clone?

Patch 1 adds BR2_PRIMARY_SITE_ONLY_EXTENDED_DOMAINS and is useful independently
from source-check. It addresses the problem where BR2_PRIMARY_SITE_ONLY cannot
be used if you also have packages under version control.

Patch 2 adds the source-check script.

Best regards,
Thomas

Thomas De Schampheleire (2):
core: add BR2_PRIMARY_SITE_ONLY_EXTENDED_DOMAINS utils/source-check: new script

Quoted 10 lines
 Config.in               |  11 +++
 package/pkg-download.mk |   8 +-
 utils/source-check      | 170 ++++++++++++++++++++++++++++++++++++++++
 3 files changed, 188 insertions(+), 1 deletion(-)
 create mode 100755 utils/source-check

--
2.26.2

I would like to ping this series for feedback:

http://patchwork.ozlabs.org/project/buildroot/patch/20200504111551.15920-2-patrickdepinguin at gmail.com/
http://patchwork.ozlabs.org/project/buildroot/patch/20200504111551.15920-3-patrickdepinguin at gmail.com/ -------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20200804/04c9525c/attachment.html>