scripts/install-buildtools: improvements
PATCH 1/4 scripts/install-buildtools: improvements
* Install directory defaults to scripts/../buildtools e.g. --directory is set by default
This avoids the user having to type in their sudo password to install in /opt/poky/<installer-version>
* Use "." rather than "source" for sourcing the environment script as not all distros (e.g. Debian) have "source" by default.
* Add buildtools/ to .gitignore
* Fix typos in example usage (--install-version -> --installer-version)
[YOCTO #13832]
--- .gitignore | 1 + scripts/install-buildtools | 13 +++++++++---- 2 files changed, 10 insertions(+), 4 deletions(-)
.gitignore +1 -0
@@ -9,6 +9,7 @@ pstage/ scripts/oe-git-proxy-socks sources/ meta-*/ +buildtools/ !meta-skeleton !meta-selftest hob-image-*.bb
scripts/install-buildtools +9 -4
@@ -17,7 +17,7 @@ # $ install-buildtools \ # --base-url http://downloads.yoctoproject.org/releases/yocto \ # --release yocto-3.1_M2 \ -# --install-version 3.0+snapshot +# --installer-version 3.0+snapshot # --build-date 202000122 # # Example usage (standard buildtools from release):
@@ -29,7 +29,7 @@ # $ install-buildtools --without-extended-buildtools \ # --base-url http://downloads.yoctoproject.org/releases/yocto \ # --release yocto-3.0.2 \ -# --install-version 3.0.2 +# --installer-version 3.0.2 # import argparse
@@ -59,6 +59,7 @@ if not bitbakepath: PROGNAME = 'install-buildtools' logger = scriptutils.logger_create(PROGNAME, stream=sys.stdout) +DEFAULT_INSTALL_DIR: str = os.path.join(os.path.split(scripts_path)[0],'buildtools') DEFAULT_BASE_URL: str = 'http://downloads.yoctoproject.org/releases/yocto' DEFAULT_RELEASE: str = 'yocto-3.1_M2' DEFAULT_INSTALLER_VERSION: str = '3.0+snapshot'
@@ -66,6 +67,7 @@ DEFAULT_BUILDDATE: str = "20200122" def main(): + global DEFAULT_INSTALL_DIR global DEFAULT_BASE_URL global DEFAULT_RELEASE global DEFAULT_INSTALLER_VERSION
@@ -73,6 +75,7 @@ def main(): filename: str = "" release: str = "" buildtools_url: str = "" + install_dir: str = "" parser = argparse.ArgumentParser( description="Buildtools installation helper",
@@ -87,6 +90,7 @@ def main(): '(optional)\nRequires --url', action='store') parser.add_argument('-d', '--directory', + default=DEFAULT_INSTALL_DIR, help='directory where buildtools SDK will be installed (optional)', action='store') parser.add_argument('-r', '--release',
@@ -216,12 +220,12 @@ def main(): st = os.stat(tmpbuildtools) os.chmod(tmpbuildtools, st.st_mode | stat.S_IEXEC) logger.debug(os.stat(tmpbuildtools)) - install_dir = "/opt/poky/%s" % args.installer_version if args.directory: install_dir = args.directory ret = subprocess.call("%s -d %s -y" % (tmpbuildtools, install_dir), shell=True) else: + install_dir = "/opt/poky/%s" % args.installer_version ret = subprocess.call("%s -y" % tmpbuildtools, shell=True) if ret != 0: logger.error("Could not run buildtools installer")
@@ -238,7 +242,8 @@ def main(): tool = 'gcc' else: tool = 'tar' - proc = subprocess.run("source %s/environment-setup-x86_64-pokysdk-linux && which %s" % + logger.debug("install_dir: %s" % install_dir) + proc = subprocess.run(". %s/environment-setup-x86_64-pokysdk-linux && which %s" % (install_dir, tool), shell=True, stdout=subprocess.PIPE) which_tool = proc.stdout.decode("utf-8")
Signature · 1 line
2.24.0
PATCH 2/4 oe-buildenv-internal: python 3.5 as min version
Python 3.4 is EOL:
https://www.python.org/downloads/release/python-3410/
The last supported distro was probably CentOS-7, which has python36 available from epel-7 or scl (as rh-python36) [1]
[1] https://www.softwarecollections.org/en/scls/rhscl/rh-python36/
--- scripts/oe-buildenv-internal | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
scripts/oe-buildenv-internal +4 -4
@@ -32,12 +32,12 @@ fi # We potentially have code that doesn't parse correctly with older versions # of Python, and rather than fixing that and being eternally vigilant for # any other new feature use, just check the version here. -py_v34_check=$(python3 -c 'import sys; print(sys.version_info >= (3,4,0))') -if [ "$py_v34_check" != "True" ]; then - echo >&2 "BitBake requires Python 3.4.0 or later as 'python3'" +py_v35_check=$(python3 -c 'import sys; print(sys.version_info >= (3,5,0))') +if [ "$py_v35_check" != "True" ]; then + echo >&2 "BitBake requires Python 3.5.0 or later as 'python3'" return 1 fi -unset py_v34_check +unset py_v35_check if [ -z "$BDIR" ]; then if [ -z "$1" ]; then
Signature · 1 line
2.24.0
PATCH 3/4 sanity.bbclass: recommend using install-buildtools
In reply to Tim OrlingFor old tar versions (1.24 and 1.28), recommned using scripts/install-buildtools
--- meta/classes/sanity.bbclass | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
meta/classes/sanity.bbclass +2 -2
@@ -533,9 +533,9 @@ def check_tar_version(sanity_data): return "Unable to execute tar --version, exit code %d\n%s\n" % (e.returncode, e.output) version = result.split()[3] if LooseVersion(version) < LooseVersion("1.24"): - return "Your version of tar is older than 1.24 and has bugs which will break builds. Please install a newer version of tar (1.28+).\n" + return "Your version of tar is older than 1.24 and has bugs which will break builds. Please install a newer version of tar (1.28+). You could use the project's buildtools-tarball from our last release or use scripts/install-buildtools\n" if LooseVersion(version) < LooseVersion("1.28"): - return "Your version of tar is older than 1.28 and does not have the support needed to enable reproducible builds. Please install a newer version of tar (you could use the projects buildtools-tarball from our last release).\n" + return "Your version of tar is older than 1.28 and does not have the support needed to enable reproducible builds. Please install a newer version of tar (you could use the projects buildtools-tarball from our last release or use scripts/install-buildtools).\n" return None # We use git parameters and functionality only found in 1.7.8 or later
Signature · 1 line
2.24.0
Re: [OE-core] [PATCH 3/4] sanity.bbclass: recommend using install-buildtools
Quoted 2 lines · Tim Orling
For old tar versions (1.24 and 1.28), recommned using
Typo.
Quoted 12 lines
scripts/install-buildtoolsSigned-off-by Tim Orling <[email protected]>--- meta/classes/sanity.bbclass | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)meta/classes/sanity.bbclass +0
-0
meta/classes/sanity.bbclass def check_tar_version(sanity_data): · line 539
Quoted 9 lines
meta/classes/sanity.bbclass +2
-2@@ -533,9 +533,9 @@ def check_tar_version(sanity_data): return "Unable to execute tar --version, exit code %d\n%s\n" % (e.returncode, e.output) version = result.split()[3] if LooseVersion(version) < LooseVersion("1.24"): - return "Your version of tar is older than 1.24 and has bugs which will break builds. Please install a newer version of tar (1.28+).\n" + return "Your version of tar is older than 1.24 and has bugs which will break builds. Please install a newer version of tar (1.28+). You could use the project's buildtools-tarball from our last release or use scripts/install-buildtools\n" if LooseVersion(version) < LooseVersion("1.28"): - return "Your version of tar is older than 1.28 and does not have the support needed to enable reproducible builds. Please install a newer version of tar (you could use the projects buildtools-tarball from our last release).\n" + return "Your version of tar is older than 1.28 and does not have the support needed to enable reproducible builds. Please install a newer version of tar (you could use the projects buildtools-tarball from our last release or use scripts/install-buildtools).\n" return None
Is it useful to keep two levels of version check?
Quoted 5 lines
meta/classes/sanity.bbclass +0
-1# We use git parameters and functionality only found in 1.7.8 or later --2.24.0
PATCH 4/4 sanity.bbclass: add test for gcc < 5.0
In reply to Tim OrlingIt is known that the version of gcc in CentOS-7 (4.8.5) causes builds to fail.
Add a test for BUILD_CC == 'gcc' and gcc < 5.0 and recommend using scripts/install-buildtools or user built buildtools-extended-tarball.
NOTE: another solution is to install devtoolset-6+ from scl [1], but this is a rather large install (> 1 Gb) and fairly invasive.
[1] https://www.softwarecollections.org/en/scls/rhscl/devtoolset-6/
--- meta/classes/sanity.bbclass | 26 +++++++++++++++++++++++++- 1 file changed, 25 insertions(+), 1 deletion(-)
meta/classes/sanity.bbclass +25 -1
@@ -521,6 +521,29 @@ def check_wsl(d): return "OpenEmbedded doesn't work under WSL at this time, sorry" return None +# The gcc version in CentOS-7 (4.8.5) is known to be a problem. +# Require at least gcc version 5.0. +# +# This can be fixed on CentOS-7 with devtoolset-6+ +# https://www.softwarecollections.org/en/scls/rhscl/devtoolset-6/ +# +# A less invasive fix is with scripts/install-buildtools (or with user +# built buildtools-extended-tarball) +# +def check_gcc_version(sanity_data): + from distutils.version import LooseVersion + import subprocess + build_cc = sanity_data.getVar('BUILD_CC').strip() + if build_cc == "gcc": + try: + result = subprocess.check_output(["gcc", "--version"], stderr=subprocess.DEVNULL).decode('utf-8') + except subprocess.CalledProcessError as e: + return "Unable to execute gcc --version, exit code %d\n%s\n" % (e.returncode, e.output) + version = result.split()[2] + if LooseVersion(version) < LooseVersion("5.0"): + return "Your version of gcc is older than 5.0 and will break builds. Please install a newer version of gcc (you could use the project's buildtools-extended-tarball or use scripts/install-buildtools).\n" + return None + # Tar version 1.24 and onwards handle overwriting symlinks correctly # but earlier versions do not; this needs to work properly for sstate # Version 1.28 is needed so opkg-build works correctly when reproducibile builds are enabled
@@ -535,7 +558,7 @@ def check_tar_version(sanity_data): if LooseVersion(version) < LooseVersion("1.24"): return "Your version of tar is older than 1.24 and has bugs which will break builds. Please install a newer version of tar (1.28+). You could use the project's buildtools-tarball from our last release or use scripts/install-buildtools\n" if LooseVersion(version) < LooseVersion("1.28"): - return "Your version of tar is older than 1.28 and does not have the support needed to enable reproducible builds. Please install a newer version of tar (you could use the projects buildtools-tarball from our last release or use scripts/install-buildtools).\n" + return "Your version of tar is older than 1.28 and does not have the support needed to enable reproducible builds. Please install a newer version of tar (you could use the project's buildtools-tarball from our last release or use scripts/install-buildtools).\n" return None # We use git parameters and functionality only found in 1.7.8 or later
@@ -634,6 +657,7 @@ def check_sanity_version_change(status, d): except ImportError as e: status.addresult('Your Python 3 is not a full install. Please install the module %s (see the Getting Started guide for further information).\n' % e.name) + status.addresult(check_gcc_version(d)) status.addresult(check_make_version(d)) status.addresult(check_patch_version(d)) status.addresult(check_tar_version(d))
Signature · 1 line
2.24.0
Re: [OE-core] [PATCH 4/4] sanity.bbclass: add test for gcc < 5.0
Quoted 12 lines · Tim Orling
meta/classes/sanity.bbclass +12
-0+# The gcc version in CentOS-7 (4.8.5) is known to be a problem. +# Require at least gcc version 5.0. +# +# This can be fixed on CentOS-7 with devtoolset-6+ +# https://www.softwarecollections.org/en/scls/rhscl/devtoolset-6/ +# +# A less invasive fix is with scripts/install-buildtools (or with user +# built buildtools-extended-tarball) +# +def check_gcc_version(sanity_data): + from distutils.version import LooseVersion + import subprocess
meta/classes/sanity.bbclass def check_wsl(d): · line 545
Quoted 10 lines
meta/classes/sanity.bbclass +10
-0+ build_cc = sanity_data.getVar('BUILD_CC').strip() + if build_cc == "gcc": + try: + result = subprocess.check_output(["gcc", "--version"], stderr=subprocess.DEVNULL).decode('utf-8') + except subprocess.CalledProcessError as e: + return "Unable to execute gcc --version, exit code %d\n%s\n" % (e.returncode, e.output) + version = result.split()[2] + if LooseVersion(version) < LooseVersion("5.0"): + return "Your version of gcc is older than 5.0 and will break builds. Please install a newer version of gcc (you could use the project's buildtools-extended-tarball or use scripts/install-buildtools).\n" + return None
I had forgotten we had this but a recent patch reminded me, would we want to tweak this function:
http://git.yoctoproject.org/cgit.cgi/poky/tree/meta/lib/oe/utils.py?h=master-next#n376
?
Cheers,
Richard
Quoted 56 lines · Tim Orling
* Install directory defaults to
scripts/../buildtoolse.g. --directory is set by default
This avoids the user having to type in their sudo password to install in /opt/poky/<installer-version>* Use "." rather than "source" for sourcing the environment script as not all distros (e.g. Debian) have "source" by default.
* Add buildtools/ to .gitignore
* Fix typos in example usage (--install-version -> --installer-version)
[YOCTO #13832]
Signed-off-by Tim Orling <[email protected]>--- .gitignore | 1 + scripts/install-buildtools | 13 +++++++++---- 2 files changed, 10 insertions(+), 4 deletions(-).gitignore +1
-0@@ -9,6 +9,7 @@ pstage/ scripts/oe-git-proxy-socks sources/ meta-*/ +buildtools/ !meta-skeleton !meta-selftest hob-image-*.bbscripts/install-buildtools +2
-2@@ -17,7 +17,7 @@ # $ install-buildtools \ # --base-url http://downloads.yoctoproject.org/releases/yocto \ # --release yocto-3.1_M2 \ -# --install-version 3.0+snapshot +# --installer-version 3.0+snapshot # --build-date 202000122 # # Example usage (standard buildtools from release):@@ -29,7 +29,7 @@ # $ install-buildtools --without-extended-buildtools \ # --base-url http://downloads.yoctoproject.org/releases/yocto \ # --release yocto-3.0.2 \ -# --install-version 3.0.2 +# --installer-version 3.0.2 # import argparse
scripts/install-buildtools if not bitbakepath: · line 62
Quoted 4 lines
scripts/install-buildtools +1
-0@@ -59,6 +59,7 @@ if not bitbakepath: PROGNAME = 'install-buildtools' logger = scriptutils.logger_create(PROGNAME, stream=sys.stdout) +DEFAULT_INSTALL_DIR: str = os.path.join(os.path.split(scripts_path)[0],'buildtools')
It looks like there are some assumptions about the minimum version of python3 on the host in order to even be able to run install-buildtools. Should that be checked explicitly?
Quoted 55 lines
scripts/install-buildtools +6
-3DEFAULT_BASE_URL: str = 'http://downloads.yoctoproject.org/releases/yocto' DEFAULT_RELEASE: str = 'yocto-3.1_M2' DEFAULT_INSTALLER_VERSION: str = '3.0+snapshot'@@ -66,6 +67,7 @@ DEFAULT_BUILDDATE: str = "20200122" def main(): + global DEFAULT_INSTALL_DIR global DEFAULT_BASE_URL global DEFAULT_RELEASE global DEFAULT_INSTALLER_VERSION@@ -73,6 +75,7 @@ def main(): filename: str = "" release: str = "" buildtools_url: str = "" + install_dir: str = "" parser = argparse.ArgumentParser( description="Buildtools installation helper",@@ -87,6 +90,7 @@ def main(): '(optional)\nRequires --url', action='store') parser.add_argument('-d', '--directory', + default=DEFAULT_INSTALL_DIR, help='directory where buildtools SDK will be installed (optional)', action='store') parser.add_argument('-r', '--release',@@ -216,12 +220,12 @@ def main(): st = os.stat(tmpbuildtools) os.chmod(tmpbuildtools, st.st_mode | stat.S_IEXEC) logger.debug(os.stat(tmpbuildtools)) - install_dir = "/opt/poky/%s" % args.installer_version if args.directory: install_dir = args.directory ret = subprocess.call("%s -d %s -y" % (tmpbuildtools, install_dir), shell=True) else: + install_dir = "/opt/poky/%s" % args.installer_version ret = subprocess.call("%s -y" % tmpbuildtools, shell=True) if ret != 0: logger.error("Could not run buildtools installer")@@ -238,7 +242,8 @@ def main(): tool = 'gcc' else: tool = 'tar' - proc = subprocess.run("source %s/environment-setup-x86_64-pokysdk-linux && which %s" % + logger.debug("install_dir: %s" % install_dir) + proc = subprocess.run(". %s/environment-setup-x86_64-pokysdk-linux && which %s" % (install_dir, tool), shell=True, stdout=subprocess.PIPE) which_tool = proc.stdout.decode("utf-8") --2.24.0
Quoted 63 lines · Andre McCurdy
Quoted 38 lines · Tim Orling
* Install directory defaults to
scripts/../buildtoolse.g. --directory is set by default
This avoids the user having to type in their sudo password to install in /opt/poky/<installer-version>* Use "." rather than "source" for sourcing the environment script as not all distros (e.g. Debian) have "source" by default.
* Add buildtools/ to .gitignore
* Fix typos in example usage (--install-version -> --installer-version)
[YOCTO #13832]
Signed-off-by Tim Orling <[email protected]>--- .gitignore | 1 + scripts/install-buildtools | 13 +++++++++---- 2 files changed, 10 insertions(+), 4 deletions(-).gitignore +1
-0@@ -9,6 +9,7 @@ pstage/ scripts/oe-git-proxy-socks sources/ meta-*/ +buildtools/ !meta-skeleton !meta-selftest hob-image-*.bbscripts/install-buildtools +0
-0scripts/install-buildtools line 18
Quoted 2 lines
scripts/install-buildtools +0
-0@@ -17,7 +17,7 @@ # $ install-buildtools \ # --base-url http://downloads.yoctoproject.org/releases/yocto\
scripts/install-buildtools line 30
Quoted 9 lines
scripts/install-buildtools +1
-1# --release yocto-3.1_M2 \ -# --install-version 3.0+snapshot +# --installer-version 3.0+snapshot # --build-date 202000122 # # Example usage (standard buildtools from release):@@ -29,7 +29,7 @@ # $ install-buildtools --without-extended-buildtools \ # --base-url http://downloads.yoctoproject.org/releases/yocto\
Quoted 10 lines
scripts/install-buildtools +1
-1# --release yocto-3.0.2 \ -# --install-version 3.0.2 +# --installer-version 3.0.2 # import argparse@@ -59,6 +59,7 @@ if not bitbakepath: PROGNAME = 'install-buildtools' logger = scriptutils.logger_create(PROGNAME, stream=sys.stdout)
Quoted 7 lines
scripts/install-buildtools if not bitbakepath: · line 62
Quoted 1 line
scripts/install-buildtools +1
-0+DEFAULT_INSTALL_DIR: str =os.path.join(os.path.split(scripts_path)[0],'buildtools')
It looks like there are some assumptions about the minimum version of python3 on the host in order to even be able to run install-buildtools. Should that be checked explicitly?
To be honest, everything was written on CentOS-7, where ‘python3’ is now python36.
The remaining supported holdout for 3.4 is Debian-8 (Jessie). Which also happens to have tar 1.27. At least it has git 2.1.4, so it’s not a complete mess.
Python 3.4 is already EOL. We should not support it for the complete build system, but I can see that the install-buildtools script should ideally still run on python34. The whole point is to make it easier to install the pre-built buildtools.
Debian-8 is EOL in June 2020.
Quoted 62 lines
scripts/install-buildtools if not bitbakepath: · line 63
Quoted 1 line
scripts/install-buildtools +0
-0DEFAULT_BASE_URL: str = 'Quoted 19 lines
scripts/install-buildtools +2
-0DEFAULT_RELEASE: str = 'yocto-3.1_M2' DEFAULT_INSTALLER_VERSION: str = '3.0+snapshot'@@ -66,6 +67,7 @@ DEFAULT_BUILDDATE: str = "20200122" def main(): + global DEFAULT_INSTALL_DIR global DEFAULT_BASE_URL global DEFAULT_RELEASE global DEFAULT_INSTALLER_VERSION@@ -73,6 +75,7 @@ def main(): filename: str = "" release: str = "" buildtools_url: str = "" + install_dir: str = "" parser = argparse.ArgumentParser( description="Buildtools installation helper",scripts/install-buildtools def main(): · line 94
Quoted 5 lines
scripts/install-buildtools +1
-0@@ -87,6 +90,7 @@ def main(): '(optional)\nRequires --url', action='store') parser.add_argument('-d', '--directory', + default=DEFAULT_INSTALL_DIR, help='directory where buildtools SDK will beinstalled (optional)',
scripts/install-buildtools def main(): · line 226
Quoted 11 lines
scripts/install-buildtools +0
-1action='store') parser.add_argument('-r', '--release',@@ -216,12 +220,12 @@ def main(): st = os.stat(tmpbuildtools) os.chmod(tmpbuildtools, st.st_mode | stat.S_IEXEC) logger.debug(os.stat(tmpbuildtools)) - install_dir = "/opt/poky/%s" % args.installer_version if args.directory: install_dir = args.directory ret = subprocess.call("%s -d %s -y" % (tmpbuildtools, install_dir),shell=True)
scripts/install-buildtools def main(): · line 241
Quoted 10 lines
scripts/install-buildtools +1
-1else: + install_dir = "/opt/poky/%s" % args.installer_version ret = subprocess.call("%s -y" % tmpbuildtools, shell=True) if ret != 0: logger.error("Could not run buildtools installer")@@ -238,7 +242,8 @@ def main(): tool = 'gcc' else: tool = 'tar' - proc = subprocess.run("source%s/environment-setup-x86_64-pokysdk-linux && which %s" %
scripts/install-buildtools def main(): · line 246
Quoted 2 lines
scripts/install-buildtools +2
-0+ logger.debug("install_dir: %s" % install_dir) + proc = subprocess.run(".%s/environment-setup-x86_64-pokysdk-linux && which %s" %
Quoted 7 lines
scripts/install-buildtools +0
-1(install_dir, tool), shell=True, stdout=subprocess.PIPE) which_tool = proc.stdout.decode("utf-8") --2.24.0
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 78 lines · Tim Orling via lists.openembedded.org
Quoted 63 lines · Andre McCurdy
Quoted 38 lines · Tim Orling
* Install directory defaults to
scripts/../buildtoolse.g. --directory is set by default
This avoids the user having to type in their sudo password to install in /opt/poky/<installer-version>* Use "." rather than "source" for sourcing the environment script as not all distros (e.g. Debian) have "source" by default.
* Add buildtools/ to .gitignore
* Fix typos in example usage (--install-version -> --installer-version)
[YOCTO #13832]
Signed-off-by Tim Orling <[email protected]>--- .gitignore | 1 + scripts/install-buildtools | 13 +++++++++---- 2 files changed, 10 insertions(+), 4 deletions(-).gitignore +1
-0@@ -9,6 +9,7 @@ pstage/ scripts/oe-git-proxy-socks sources/ meta-*/ +buildtools/ !meta-skeleton !meta-selftest hob-image-*.bbscripts/install-buildtools +0
-0scripts/install-buildtools line 18
Quoted 2 lines
scripts/install-buildtools +0
-0@@ -17,7 +17,7 @@ # $ install-buildtools \ # --base-url http://downloads.yoctoproject.org/releases/yocto\
scripts/install-buildtools line 30
Quoted 9 lines
scripts/install-buildtools +1
-1# --release yocto-3.1_M2 \ -# --install-version 3.0+snapshot +# --installer-version 3.0+snapshot # --build-date 202000122 # # Example usage (standard buildtools from release):@@ -29,7 +29,7 @@ # $ install-buildtools --without-extended-buildtools \ # --base-url http://downloads.yoctoproject.org/releases/yocto\
Quoted 10 lines
scripts/install-buildtools +1
-1# --release yocto-3.0.2 \ -# --install-version 3.0.2 +# --installer-version 3.0.2 # import argparse@@ -59,6 +59,7 @@ if not bitbakepath: PROGNAME = 'install-buildtools' logger = scriptutils.logger_create(PROGNAME, stream=sys.stdout)Quoted 7 lines
scripts/install-buildtools if not bitbakepath: · line 62
Quoted 1 line
scripts/install-buildtools +1
-0+DEFAULT_INSTALL_DIR: str =os.path.join(os.path.split(scripts_path)[0],'buildtools')
It looks like there are some assumptions about the minimum version of python3 on the host in order to even be able to run install-buildtools. Should that be checked explicitly?
To be honest, everything was written on CentOS-7, where ‘python3’ is now python36.
Quoted 9 lines
The remaining supported holdout for 3.4 is Debian-8 (Jessie). Which also happens to have tar 1.27. At least it has git 2.1.4, so it’s not a complete mess.
Python 3.4 is already EOL. We should not support it for the complete build system, but I can see that the install-buildtools script should ideally still run on python34. The whole point is to make it easier to install the pre-built buildtools.
Python 3.5 is now required in bitbake/lib/bb/__init__.py Which is needed for bb.utils.md5sum_file and sha256_file to check the checksum of the download. I’d rather not have to rewrite those functions.
This means Debian-8 is a conundrum.
Quoted 66 lines
Debian-8 is EOL in June 2020.
Quoted 62 lines
scripts/install-buildtools if not bitbakepath: · line 63
Quoted 1 line
scripts/install-buildtools +0
-0DEFAULT_BASE_URL: str = 'Quoted 19 lines
scripts/install-buildtools +2
-0DEFAULT_RELEASE: str = 'yocto-3.1_M2' DEFAULT_INSTALLER_VERSION: str = '3.0+snapshot'@@ -66,6 +67,7 @@ DEFAULT_BUILDDATE: str = "20200122" def main(): + global DEFAULT_INSTALL_DIR global DEFAULT_BASE_URL global DEFAULT_RELEASE global DEFAULT_INSTALLER_VERSION@@ -73,6 +75,7 @@ def main(): filename: str = "" release: str = "" buildtools_url: str = "" + install_dir: str = "" parser = argparse.ArgumentParser( description="Buildtools installation helper",scripts/install-buildtools def main(): · line 94
Quoted 5 lines
scripts/install-buildtools +1
-0@@ -87,6 +90,7 @@ def main(): '(optional)\nRequires --url', action='store') parser.add_argument('-d', '--directory', + default=DEFAULT_INSTALL_DIR, help='directory where buildtools SDK will beinstalled (optional)',
scripts/install-buildtools def main(): · line 226
Quoted 11 lines
scripts/install-buildtools +0
-1action='store') parser.add_argument('-r', '--release',@@ -216,12 +220,12 @@ def main(): st = os.stat(tmpbuildtools) os.chmod(tmpbuildtools, st.st_mode | stat.S_IEXEC) logger.debug(os.stat(tmpbuildtools)) - install_dir = "/opt/poky/%s" % args.installer_version if args.directory: install_dir = args.directory ret = subprocess.call("%s -d %s -y" % (tmpbuildtools, install_dir),shell=True)
scripts/install-buildtools def main(): · line 241
Quoted 10 lines
scripts/install-buildtools +1
-1else: + install_dir = "/opt/poky/%s" % args.installer_version ret = subprocess.call("%s -y" % tmpbuildtools, shell=True) if ret != 0: logger.error("Could not run buildtools installer")@@ -238,7 +242,8 @@ def main(): tool = 'gcc' else: tool = 'tar' - proc = subprocess.run("source%s/environment-setup-x86_64-pokysdk-linux && which %s" %
scripts/install-buildtools def main(): · line 246
Quoted 2 lines
scripts/install-buildtools +2
-0+ logger.debug("install_dir: %s" % install_dir) + proc = subprocess.run(".%s/environment-setup-x86_64-pokysdk-linux && which %s" %
Quoted 7 lines
scripts/install-buildtools +0
-1(install_dir, tool), shell=True, stdout=subprocess.PIPE) which_tool = proc.stdout.decode("utf-8") --2.24.0
Quoted 71 lines · Tim Orling
Quoted 70 lines · Tim Orling via lists.openembedded.org
Quoted 67 lines · Andre McCurdy
Quoted 56 lines · Tim Orling
* Install directory defaults to
scripts/../buildtoolse.g. --directory is set by default
This avoids the user having to type in their sudo password to install in /opt/poky/<installer-version>* Use "." rather than "source" for sourcing the environment script as not all distros (e.g. Debian) have "source" by default.
* Add buildtools/ to .gitignore
* Fix typos in example usage (--install-version -> --installer-version)
[YOCTO #13832]
Signed-off-by Tim Orling <[email protected]>--- .gitignore | 1 + scripts/install-buildtools | 13 +++++++++---- 2 files changed, 10 insertions(+), 4 deletions(-).gitignore +1
-0@@ -9,6 +9,7 @@ pstage/ scripts/oe-git-proxy-socks sources/ meta-*/ +buildtools/ !meta-skeleton !meta-selftest hob-image-*.bbscripts/install-buildtools +2
-2@@ -17,7 +17,7 @@ # $ install-buildtools \ # --base-url http://downloads.yoctoproject.org/releases/yocto \ # --release yocto-3.1_M2 \ -# --install-version 3.0+snapshot +# --installer-version 3.0+snapshot # --build-date 202000122 # # Example usage (standard buildtools from release):@@ -29,7 +29,7 @@ # $ install-buildtools --without-extended-buildtools \ # --base-url http://downloads.yoctoproject.org/releases/yocto \ # --release yocto-3.0.2 \ -# --install-version 3.0.2 +# --installer-version 3.0.2 # import argparsescripts/install-buildtools if not bitbakepath: · line 62
Quoted 4 lines
scripts/install-buildtools +1
-0@@ -59,6 +59,7 @@ if not bitbakepath: PROGNAME = 'install-buildtools' logger = scriptutils.logger_create(PROGNAME, stream=sys.stdout) +DEFAULT_INSTALL_DIR: str = os.path.join(os.path.split(scripts_path)[0],'buildtools')It looks like there are some assumptions about the minimum version of python3 on the host in order to even be able to run install-buildtools. Should that be checked explicitly?
Quoted 8 lines
Quoted 5 lines
To be honest, everything was written on CentOS-7, where ‘python3’ is now python36.
The remaining supported holdout for 3.4 is Debian-8 (Jessie). Which also happens to have tar 1.27. At least it has git 2.1.4, so it’s not a complete mess.
Python 3.4 is already EOL. We should not support it for the complete build system, but I can see that the install-buildtools script should ideally still run on python34. The whole point is to make it easier to install the pre-built buildtools.
Python 3.5 is now required in
bitbake/lib/bb/__init__.py
Which is needed for bb.utils.md5sum_file andsha256_fileto check the checksum of the download. I’d rather not have to rewrite those functions.
Looks like copy and pasting ~30 lines? Maybe not even much of a net increase in code size if you can remove the existing code to check that bitbake libs are available...
def _hasher(method, filename):
import mmap
with open(filename, "rb") as f:
try:
with mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) as mm:
for chunk in iter(lambda: mm.read(8192), b''):
method.update(chunk)
except ValueError:
# You can't mmap() an empty file so silence this exception pass
return method.hexdigest()
def md5_file(filename):
"""
Return the hex string representation of the MD5 checksum of filename. """
import hashlib
return _hasher(hashlib.md5(), filename)
def sha256_file(filename):
"""
Return the hex string representation of the 256-bit SHA checksum of filename.
"""
import hashlib
return _hasher(hashlib.sha256(), filename)
Quoted 60 lines
Quoted 60 lines
Debian-8 is EOL in June 2020.
Quoted 57 lines
Quoted 56 lines
scripts/install-buildtools +6
-3DEFAULT_BASE_URL: str = 'http://downloads.yoctoproject.org/releases/yocto' DEFAULT_RELEASE: str = 'yocto-3.1_M2' DEFAULT_INSTALLER_VERSION: str = '3.0+snapshot'@@ -66,6 +67,7 @@ DEFAULT_BUILDDATE: str = "20200122" def main(): + global DEFAULT_INSTALL_DIR global DEFAULT_BASE_URL global DEFAULT_RELEASE global DEFAULT_INSTALLER_VERSION@@ -73,6 +75,7 @@ def main(): filename: str = "" release: str = "" buildtools_url: str = "" + install_dir: str = "" parser = argparse.ArgumentParser( description="Buildtools installation helper",@@ -87,6 +90,7 @@ def main(): '(optional)\nRequires --url', action='store') parser.add_argument('-d', '--directory', + default=DEFAULT_INSTALL_DIR, help='directory where buildtools SDK will be installed (optional)', action='store') parser.add_argument('-r', '--release',@@ -216,12 +220,12 @@ def main(): st = os.stat(tmpbuildtools) os.chmod(tmpbuildtools, st.st_mode | stat.S_IEXEC) logger.debug(os.stat(tmpbuildtools)) - install_dir = "/opt/poky/%s" % args.installer_version if args.directory: install_dir = args.directory ret = subprocess.call("%s -d %s -y" % (tmpbuildtools, install_dir), shell=True) else: + install_dir = "/opt/poky/%s" % args.installer_version ret = subprocess.call("%s -y" % tmpbuildtools, shell=True) if ret != 0: logger.error("Could not run buildtools installer")@@ -238,7 +242,8 @@ def main(): tool = 'gcc' else: tool = 'tar' - proc = subprocess.run("source %s/environment-setup-x86_64-pokysdk-linux && which %s" % + logger.debug("install_dir: %s" % install_dir) + proc = subprocess.run(". %s/environment-setup-x86_64-pokysdk-linux && which %s" % (install_dir, tool), shell=True, stdout=subprocess.PIPE) which_tool = proc.stdout.decode("utf-8") --2.24.0
Quoted 55 lines · Andre McCurdy
Quoted 22 lines · Tim Orling
Quoted 10 lines · Tim Orling via lists.openembedded.org
On Wed, Apr 1, 2020 at 1:09 PM Andre McCurdy <[email protected]
Quoted 5 lines
wrote:
It looks like there are some assumptions about the minimum version of
python3 on the host in order to even be able to run install-buildtools. Should that be checked explicitly?To be honest, everything was written on CentOS-7, where ‘python3’ is now python36.
Quoted 8 lines
The remaining supported holdout for 3.4 is Debian-8 (Jessie). Which also happens to have tar 1.27. At least it has git 2.1.4, so it’s not a complete mess.
Python 3.4 is already EOL. We should not support it for the complete build system, but I can see that the install-buildtools script should ideally still run on python34. The whole point is to make it easier to install the pre-built buildtools.
Quoted 4 lines
Python 3.5 is now required in
bitbake/lib/bb/__init__.pyWhich is needed for bb.utils.md5sum_file andsha256_fileto check the checksum of the download. I’d rather not have to rewrite those functions.Looks like copy and pasting ~30 lines? Maybe not even much of a net increase in code size if you can remove the existing code to check that bitbake libs are available...
def
_hasher(method, filename):
import mmapwith
open(filename, "rb")as f:
try:
withmmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ)as
mm:
for chunk initer(lambda: mm.read(8192), b''):
method.update(chunk)
except ValueError:
# You can'tmmap()an empty file so silence this
exception
pass
returnmethod.hexdigest()def
md5_file(filename):
"""
Return the hex string representation of the MD5 checksum of
filename.
"""
import hashlib
return_hasher(hashlib.md5(), filename)
Quoted 8 lines
def
sha256_file(filename):
"""
Return the hex string representation of the 256-bit SHA checksum
of
filename.
"""
import hashlib
return_hasher(hashlib.sha256(), filename)
Can you send a patch against master-next please? The buildtools installer is one place we probably do want to support older pythons and it shouldn't be too onnerous.
Cheers,
Richard
Quoted 69 lines · Richard Purdie
Quoted 55 lines · Andre McCurdy
Quoted 22 lines · Tim Orling
Quoted 10 lines · Tim Orling via lists.openembedded.org
On Wed, Apr 1, 2020 at 1:09 PM Andre McCurdy <[email protected]
Quoted 5 lines
wrote:
It looks like there are some assumptions about the minimum version of
python3 on the host in order to even be able to run install-buildtools. Should that be checked explicitly?To be honest, everything was written on CentOS-7, where ‘python3’ is now python36.
Quoted 8 lines
The remaining supported holdout for 3.4 is Debian-8 (Jessie). Which also happens to have tar 1.27. At least it has git 2.1.4, so it’s not a complete mess.
Python 3.4 is already EOL. We should not support it for the complete build system, but I can see that the install-buildtools script should ideally still run on python34. The whole point is to make it easier to install the pre-built buildtools.
Quoted 4 lines
Python 3.5 is now required in
bitbake/lib/bb/__init__.pyWhich is needed for bb.utils.md5sum_file andsha256_fileto check the checksum of the download. I’d rather not have to rewrite those functions.Looks like copy and pasting ~30 lines? Maybe not even much of a net increase in code size if you can remove the existing code to check that bitbake libs are available...
def
_hasher(method, filename):
import mmapwith
open(filename, "rb")as f:
try:
withmmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ)as
mm:
for chunk initer(lambda: mm.read(8192), b''):
method.update(chunk)
except ValueError:
# You can'tmmap()an empty file so silence this
exception
pass
returnmethod.hexdigest()def
md5_file(filename):
"""
Return the hex string representation of the MD5 checksum of
filename.
"""
import hashlib
return_hasher(hashlib.md5(), filename)Quoted 8 lines
def
sha256_file(filename):
"""
Return the hex string representation of the 256-bit SHA checksum
of
filename.
"""
import hashlib
return_hasher(hashlib.sha256(), filename)
Quoted 4 lines
Can you send a patch against master-next please? The buildtools installer is one place we probably do want to support older pythons and it shouldn't be too onnerous.
Submitted. The check for whether the install worked or not was the hardest part. After many iterations, I just could not get subprocess.Popen to work as expected on python 3.4.
https://patchwork.openembedded.org/patch/171593/ <https://patchwork.openembedded.org/patch/171593/>
Thank you Andre for the testing and input. It takes a village to make a good release.
Quoted 5 lines
Cheers,
Richard