Skip to contentSkip to conversation

scripts/install-buildtools: improvements

PATCH Superseded Tim Orling 4 participants 12 messages Mar 30, 2020 – Apr 3, 2020 1 message not in the archive

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

Tim Orling

author
#0 lore raw

PATCH 1/4 scripts/install-buildtools: improvements

* Install directory defaults to scripts/../buildtools e.g. --directory is set by default
This avoids the user having to type in their sudo password to install in /opt/poky/<installer-version>

* Use "." rather than "source" for sourcing the environment script as not all distros (e.g. Debian) have "source" by default.

* Add buildtools/ to .gitignore

* Fix typos in example usage (--install-version -> --installer-version)

[YOCTO #13832]

---
 .gitignore                 |  1 +
 scripts/install-buildtools | 13 +++++++++----
 2 files changed, 10 insertions(+), 4 deletions(-)
.gitignore +1 -0
index d0e6b2fb89..b66d371aac 100644
@@ -9,6 +9,7 @@ pstage/
 scripts/oe-git-proxy-socks
 sources/
 meta-*/
+buildtools/
 !meta-skeleton
 !meta-selftest
 hob-image-*.bb
scripts/install-buildtools +9 -4
index 0947e9c4d6..49cab1345a 100755
@@ -17,7 +17,7 @@
 #        $ install-buildtools \
 #          --base-url http://downloads.yoctoproject.org/releases/yocto \
 #          --release yocto-3.1_M2 \
-#          --install-version 3.0+snapshot
+#          --installer-version 3.0+snapshot
 #          --build-date 202000122
 #
 #  Example usage (standard buildtools from release):
@@ -29,7 +29,7 @@
 #        $ install-buildtools --without-extended-buildtools \
 #          --base-url http://downloads.yoctoproject.org/releases/yocto \
 #          --release yocto-3.0.2 \
-#          --install-version 3.0.2
+#          --installer-version 3.0.2
 #
 
 import argparse
@@ -59,6 +59,7 @@ if not bitbakepath:
 PROGNAME = 'install-buildtools'
 logger = scriptutils.logger_create(PROGNAME, stream=sys.stdout)
 
+DEFAULT_INSTALL_DIR: str = os.path.join(os.path.split(scripts_path)[0],'buildtools')
 DEFAULT_BASE_URL: str = 'http://downloads.yoctoproject.org/releases/yocto'
 DEFAULT_RELEASE: str = 'yocto-3.1_M2'
 DEFAULT_INSTALLER_VERSION: str = '3.0+snapshot'
@@ -66,6 +67,7 @@ DEFAULT_BUILDDATE: str = "20200122"
 
 
 def main():
+    global DEFAULT_INSTALL_DIR
     global DEFAULT_BASE_URL
     global DEFAULT_RELEASE
     global DEFAULT_INSTALLER_VERSION
@@ -73,6 +75,7 @@ def main():
     filename: str = ""
     release: str = ""
     buildtools_url: str = ""
+    install_dir: str = ""
 
     parser = argparse.ArgumentParser(
         description="Buildtools installation helper",
@@ -87,6 +90,7 @@ def main():
                              '(optional)\nRequires --url',
                         action='store')
     parser.add_argument('-d', '--directory',
+                        default=DEFAULT_INSTALL_DIR,
                         help='directory where buildtools SDK will be installed (optional)',
                         action='store')
     parser.add_argument('-r', '--release',
@@ -216,12 +220,12 @@ def main():
         st = os.stat(tmpbuildtools)
         os.chmod(tmpbuildtools, st.st_mode | stat.S_IEXEC)
         logger.debug(os.stat(tmpbuildtools))
-        install_dir = "/opt/poky/%s" % args.installer_version
         if args.directory:
             install_dir = args.directory
             ret = subprocess.call("%s -d %s -y" %
                                   (tmpbuildtools, install_dir), shell=True)
         else:
+            install_dir = "/opt/poky/%s" % args.installer_version
             ret = subprocess.call("%s -y" % tmpbuildtools, shell=True)
         if ret != 0:
             logger.error("Could not run buildtools installer")
@@ -238,7 +242,8 @@ def main():
             tool = 'gcc'
         else:
             tool = 'tar'
-        proc = subprocess.run("source %s/environment-setup-x86_64-pokysdk-linux && which %s" %
+        logger.debug("install_dir: %s" % install_dir)
+        proc = subprocess.run(". %s/environment-setup-x86_64-pokysdk-linux && which %s" %
                               (install_dir, tool),
                               shell=True, stdout=subprocess.PIPE)
         which_tool = proc.stdout.decode("utf-8")
Signature · 1 line
2.24.0
author
#1 lore raw

PATCH 2/4 oe-buildenv-internal: python 3.5 as min version

Python 3.4 is EOL:
https://www.python.org/downloads/release/python-3410/

The last supported distro was probably CentOS-7, which has python36 available from epel-7 or scl (as rh-python36) [1]

[1] https://www.softwarecollections.org/en/scls/rhscl/rh-python36/

---
 scripts/oe-buildenv-internal | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
scripts/oe-buildenv-internal +4 -4
index 8cbe34669d..c62688fbd2 100755
@@ -32,12 +32,12 @@ fi
 # We potentially have code that doesn't parse correctly with older versions 
 # of Python, and rather than fixing that and being eternally vigilant for 
 # any other new feature use, just check the version here.
-py_v34_check=$(python3 -c 'import sys; print(sys.version_info >= (3,4,0))')
-if [ "$py_v34_check" != "True" ]; then
-    echo >&2 "BitBake requires Python 3.4.0 or later as 'python3'"
+py_v35_check=$(python3 -c 'import sys; print(sys.version_info >= (3,5,0))')
+if [ "$py_v35_check" != "True" ]; then
+    echo >&2 "BitBake requires Python 3.5.0 or later as 'python3'"
     return 1
 fi
-unset py_v34_check
+unset py_v35_check
 
 if [ -z "$BDIR" ]; then
     if [ -z "$1" ]; then
Signature · 1 line
2.24.0
author
#2 lore raw

PATCH 3/4 sanity.bbclass: recommend using install-buildtools

In reply to Tim Orling

For old tar versions (1.24 and 1.28), recommned using scripts/install-buildtools

---
 meta/classes/sanity.bbclass | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
meta/classes/sanity.bbclass +2 -2
index cca5cdad17..0cf955a341 100644
@@ -533,9 +533,9 @@ def check_tar_version(sanity_data):
         return "Unable to execute tar --version, exit code %d\n%s\n" % (e.returncode, e.output)
     version = result.split()[3]
     if LooseVersion(version) < LooseVersion("1.24"):
-        return "Your version of tar is older than 1.24 and has bugs which will break builds. Please install a newer version of tar (1.28+).\n"
+        return "Your version of tar is older than 1.24 and has bugs which will break builds. Please install a newer version of tar (1.28+). You could use the project's buildtools-tarball from our last release or use scripts/install-buildtools\n"
     if LooseVersion(version) < LooseVersion("1.28"):
-        return "Your version of tar is older than 1.28 and does not have the support needed to enable reproducible builds. Please install a newer version of tar (you could use the projects buildtools-tarball from our last release).\n"
+        return "Your version of tar is older than 1.28 and does not have the support needed to enable reproducible builds. Please install a newer version of tar (you could use the projects buildtools-tarball from our last release or use scripts/install-buildtools).\n"
     return None
 
 # We use git parameters and functionality only found in 1.7.8 or later
Signature · 1 line
2.24.0
#3 lore raw

Re: [OE-core] [PATCH 3/4] sanity.bbclass: recommend using install-buildtools

Quoted 2 lines · Tim Orling

For old tar versions (1.24 and 1.28), recommned using

Typo.

Quoted 12 lines

scripts/install-buildtools

---
 meta/classes/sanity.bbclass | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
meta/classes/sanity.bbclass +0 -0
index cca5cdad17..0cf955a341 100644

meta/classes/sanity.bbclass def check_tar_version(sanity_data): · line 539

Quoted 9 lines
meta/classes/sanity.bbclass +2 -2
@@ -533,9 +533,9 @@ def check_tar_version(sanity_data):
         return "Unable to execute tar --version, exit code %d\n%s\n" % (e.returncode, e.output)
     version = result.split()[3]
     if LooseVersion(version) < LooseVersion("1.24"):
-        return "Your version of tar is older than 1.24 and has bugs which will break builds. Please install a newer version of tar (1.28+).\n"
+        return "Your version of tar is older than 1.24 and has bugs which will break builds. Please install a newer version of tar (1.28+). You could use the project's buildtools-tarball from our last release or use scripts/install-buildtools\n"
     if LooseVersion(version) < LooseVersion("1.28"):
-        return "Your version of tar is older than 1.28 and does not have the support needed to enable reproducible builds. Please install a newer version of tar (you could use the projects buildtools-tarball from our last release).\n"
+        return "Your version of tar is older than 1.28 and does not have the support needed to enable reproducible builds. Please install a newer version of tar (you could use the projects buildtools-tarball from our last release or use scripts/install-buildtools).\n"
     return None

Is it useful to keep two levels of version check?

Quoted 5 lines
meta/classes/sanity.bbclass +0 -1
 # We use git parameters and functionality only found in 1.7.8 or later
--

2.24.0

author
#4 lore raw

PATCH 4/4 sanity.bbclass: add test for gcc < 5.0

In reply to Tim Orling

It is known that the version of gcc in CentOS-7 (4.8.5) causes builds to fail.

Add a test for BUILD_CC == 'gcc' and gcc < 5.0 and recommend using scripts/install-buildtools or user built buildtools-extended-tarball.

NOTE: another solution is to install devtoolset-6+ from scl [1], but this is a rather large install (> 1 Gb) and fairly invasive.

[1] https://www.softwarecollections.org/en/scls/rhscl/devtoolset-6/

---
 meta/classes/sanity.bbclass | 26 +++++++++++++++++++++++++-
 1 file changed, 25 insertions(+), 1 deletion(-)
meta/classes/sanity.bbclass +25 -1
index 0cf955a341..a1cb8c6b88 100644
@@ -521,6 +521,29 @@ def check_wsl(d):
             return "OpenEmbedded doesn't work under WSL at this time, sorry"
     return None
 
+# The gcc version in CentOS-7 (4.8.5) is known to be a problem.
+# Require at least gcc version 5.0.
+#
+# This can be fixed on CentOS-7 with devtoolset-6+
+# https://www.softwarecollections.org/en/scls/rhscl/devtoolset-6/
+#
+# A less invasive fix is with scripts/install-buildtools (or with user
+# built buildtools-extended-tarball)
+#
+def check_gcc_version(sanity_data):
+    from distutils.version import LooseVersion
+    import subprocess
+    build_cc = sanity_data.getVar('BUILD_CC').strip()
+    if build_cc == "gcc":
+        try:
+            result = subprocess.check_output(["gcc", "--version"], stderr=subprocess.DEVNULL).decode('utf-8')
+        except subprocess.CalledProcessError as e:
+            return "Unable to execute gcc --version, exit code %d\n%s\n" % (e.returncode, e.output)
+        version = result.split()[2]
+        if LooseVersion(version) < LooseVersion("5.0"):
+            return "Your version of gcc is older than 5.0 and will break builds. Please install a newer version of gcc (you could use the project's buildtools-extended-tarball or use scripts/install-buildtools).\n"
+    return None
+
 # Tar version 1.24 and onwards handle overwriting symlinks correctly
 # but earlier versions do not; this needs to work properly for sstate
 # Version 1.28 is needed so opkg-build works correctly when reproducibile builds are enabled
@@ -535,7 +558,7 @@ def check_tar_version(sanity_data):
     if LooseVersion(version) < LooseVersion("1.24"):
         return "Your version of tar is older than 1.24 and has bugs which will break builds. Please install a newer version of tar (1.28+). You could use the project's buildtools-tarball from our last release or use scripts/install-buildtools\n"
     if LooseVersion(version) < LooseVersion("1.28"):
-        return "Your version of tar is older than 1.28 and does not have the support needed to enable reproducible builds. Please install a newer version of tar (you could use the projects buildtools-tarball from our last release or use scripts/install-buildtools).\n"
+        return "Your version of tar is older than 1.28 and does not have the support needed to enable reproducible builds. Please install a newer version of tar (you could use the project's buildtools-tarball from our last release or use scripts/install-buildtools).\n"
     return None
 
 # We use git parameters and functionality only found in 1.7.8 or later
@@ -634,6 +657,7 @@ def check_sanity_version_change(status, d):
     except ImportError as e:
         status.addresult('Your Python 3 is not a full install. Please install the module %s (see the Getting Started guide for further information).\n' % e.name)
 
+    status.addresult(check_gcc_version(d))
     status.addresult(check_make_version(d))
     status.addresult(check_patch_version(d))
     status.addresult(check_tar_version(d))
Signature · 1 line
2.24.0
#5 lore raw

Re: [OE-core] [PATCH 4/4] sanity.bbclass: add test for gcc < 5.0

Quoted 12 lines · Tim Orling
meta/classes/sanity.bbclass +12 -0
+# The gcc version in CentOS-7 (4.8.5) is known to be a problem.
+# Require at least gcc version 5.0.
+#
+# This can be fixed on CentOS-7 with devtoolset-6+
+# https://www.softwarecollections.org/en/scls/rhscl/devtoolset-6/
+#
+# A less invasive fix is with scripts/install-buildtools (or with user
+# built buildtools-extended-tarball)
+#
+def check_gcc_version(sanity_data):
+    from distutils.version import LooseVersion
+    import subprocess

meta/classes/sanity.bbclass def check_wsl(d): · line 545

Quoted 10 lines
meta/classes/sanity.bbclass +10 -0
+    build_cc = sanity_data.getVar('BUILD_CC').strip()
+    if build_cc == "gcc":
+        try:
+            result = subprocess.check_output(["gcc", "--version"], stderr=subprocess.DEVNULL).decode('utf-8')
+        except subprocess.CalledProcessError as e:
+            return "Unable to execute gcc --version, exit code %d\n%s\n" % (e.returncode, e.output)
+        version = result.split()[2]
+        if LooseVersion(version) < LooseVersion("5.0"):
+            return "Your version of gcc is older than 5.0 and will break builds. Please install a newer version of gcc (you could use the project's buildtools-extended-tarball or use scripts/install-buildtools).\n"
+    return None

I had forgotten we had this but a recent patch reminded me, would we want to tweak this function:

http://git.yoctoproject.org/cgit.cgi/poky/tree/meta/lib/oe/utils.py?h=master-next#n376

?

Cheers,

Richard

#6 lore raw
In reply to Tim Orling
Quoted 56 lines · Tim Orling

* Install directory defaults to scripts/../buildtools e.g. --directory is set by default
This avoids the user having to type in their sudo password to install in /opt/poky/<installer-version>

* Use "." rather than "source" for sourcing the environment script as not all distros (e.g. Debian) have "source" by default.

* Add buildtools/ to .gitignore

* Fix typos in example usage (--install-version -> --installer-version)

[YOCTO #13832]

---
 .gitignore                 |  1 +
 scripts/install-buildtools | 13 +++++++++----
 2 files changed, 10 insertions(+), 4 deletions(-)
.gitignore +1 -0
index d0e6b2fb89..b66d371aac 100644
@@ -9,6 +9,7 @@ pstage/
 scripts/oe-git-proxy-socks
 sources/
 meta-*/
+buildtools/
 !meta-skeleton
 !meta-selftest
 hob-image-*.bb
scripts/install-buildtools +2 -2
index 0947e9c4d6..49cab1345a 100755
@@ -17,7 +17,7 @@
 #        $ install-buildtools \
 #          --base-url http://downloads.yoctoproject.org/releases/yocto \
 #          --release yocto-3.1_M2 \
-#          --install-version 3.0+snapshot
+#          --installer-version 3.0+snapshot
 #          --build-date 202000122
 #
 #  Example usage (standard buildtools from release):
@@ -29,7 +29,7 @@
 #        $ install-buildtools --without-extended-buildtools \
 #          --base-url http://downloads.yoctoproject.org/releases/yocto \
 #          --release yocto-3.0.2 \
-#          --install-version 3.0.2
+#          --installer-version 3.0.2
 #
 
 import argparse

scripts/install-buildtools if not bitbakepath: · line 62

Quoted 4 lines
scripts/install-buildtools +1 -0
@@ -59,6 +59,7 @@ if not bitbakepath:
 PROGNAME = 'install-buildtools'
 logger = scriptutils.logger_create(PROGNAME, stream=sys.stdout)
 
+DEFAULT_INSTALL_DIR: str = os.path.join(os.path.split(scripts_path)[0],'buildtools')

It looks like there are some assumptions about the minimum version of python3 on the host in order to even be able to run install-buildtools. Should that be checked explicitly?

Quoted 55 lines
scripts/install-buildtools +6 -3
 DEFAULT_BASE_URL: str = 'http://downloads.yoctoproject.org/releases/yocto'
 DEFAULT_RELEASE: str = 'yocto-3.1_M2'
 DEFAULT_INSTALLER_VERSION: str = '3.0+snapshot'
@@ -66,6 +67,7 @@ DEFAULT_BUILDDATE: str = "20200122"
 
 
 def main():
+    global DEFAULT_INSTALL_DIR
     global DEFAULT_BASE_URL
     global DEFAULT_RELEASE
     global DEFAULT_INSTALLER_VERSION
@@ -73,6 +75,7 @@ def main():
     filename: str = ""
     release: str = ""
     buildtools_url: str = ""
+    install_dir: str = ""
 
     parser = argparse.ArgumentParser(
         description="Buildtools installation helper",
@@ -87,6 +90,7 @@ def main():
                              '(optional)\nRequires --url',
                         action='store')
     parser.add_argument('-d', '--directory',
+                        default=DEFAULT_INSTALL_DIR,
                         help='directory where buildtools SDK will be installed (optional)',
                         action='store')
     parser.add_argument('-r', '--release',
@@ -216,12 +220,12 @@ def main():
         st = os.stat(tmpbuildtools)
         os.chmod(tmpbuildtools, st.st_mode | stat.S_IEXEC)
         logger.debug(os.stat(tmpbuildtools))
-        install_dir = "/opt/poky/%s" % args.installer_version
         if args.directory:
             install_dir = args.directory
             ret = subprocess.call("%s -d %s -y" %
                                   (tmpbuildtools, install_dir), shell=True)
         else:
+            install_dir = "/opt/poky/%s" % args.installer_version
             ret = subprocess.call("%s -y" % tmpbuildtools, shell=True)
         if ret != 0:
             logger.error("Could not run buildtools installer")
@@ -238,7 +242,8 @@ def main():
             tool = 'gcc'
         else:
             tool = 'tar'
-        proc = subprocess.run("source %s/environment-setup-x86_64-pokysdk-linux && which %s" %
+        logger.debug("install_dir: %s" % install_dir)
+        proc = subprocess.run(". %s/environment-setup-x86_64-pokysdk-linux && which %s" %
                               (install_dir, tool),
                               shell=True, stdout=subprocess.PIPE)
         which_tool = proc.stdout.decode("utf-8")
--

2.24.0

#7 lore raw
Quoted 63 lines · Andre McCurdy
Quoted 38 lines · Tim Orling

* Install directory defaults to scripts/../buildtools e.g. --directory is set by default
This avoids the user having to type in their sudo password to install in /opt/poky/<installer-version>

* Use "." rather than "source" for sourcing the environment script as not all distros (e.g. Debian) have "source" by default.

* Add buildtools/ to .gitignore

* Fix typos in example usage (--install-version -> --installer-version)

[YOCTO #13832]

---
 .gitignore                 |  1 +
 scripts/install-buildtools | 13 +++++++++----
 2 files changed, 10 insertions(+), 4 deletions(-)
.gitignore +1 -0
index d0e6b2fb89..b66d371aac 100644
@@ -9,6 +9,7 @@ pstage/
 scripts/oe-git-proxy-socks
 sources/
 meta-*/
+buildtools/
 !meta-skeleton
 !meta-selftest
 hob-image-*.bb
scripts/install-buildtools +0 -0
index 0947e9c4d6..49cab1345a 100755

scripts/install-buildtools line 18

Quoted 2 lines
scripts/install-buildtools +0 -0
@@ -17,7 +17,7 @@
 #        $ install-buildtools \
 #          --base-url http://downloads.yoctoproject.org/releases/yocto

\

scripts/install-buildtools line 30

Quoted 9 lines
scripts/install-buildtools +1 -1
 #          --release yocto-3.1_M2 \
-#          --install-version 3.0+snapshot
+#          --installer-version 3.0+snapshot
 #          --build-date 202000122
 #
 #  Example usage (standard buildtools from release):
@@ -29,7 +29,7 @@
 #        $ install-buildtools --without-extended-buildtools \
 #          --base-url http://downloads.yoctoproject.org/releases/yocto

\

Quoted 10 lines
scripts/install-buildtools +1 -1
 #          --release yocto-3.0.2 \
-#          --install-version 3.0.2
+#          --installer-version 3.0.2
 #
 
 import argparse
@@ -59,6 +59,7 @@ if not bitbakepath:
 PROGNAME = 'install-buildtools'
 logger = scriptutils.logger_create(PROGNAME, stream=sys.stdout)
 
Quoted 7 lines

scripts/install-buildtools if not bitbakepath: · line 62

Quoted 1 line
scripts/install-buildtools +1 -0
+DEFAULT_INSTALL_DIR: str =

os.path.join(os.path.split(scripts_path)[0],'buildtools')

It looks like there are some assumptions about the minimum version of python3 on the host in order to even be able to run install-buildtools. Should that be checked explicitly?

To be honest, everything was written on CentOS-7, where ‘python3’ is now python36.

The remaining supported holdout for 3.4 is Debian-8 (Jessie). Which also happens to have tar 1.27. At least it has git 2.1.4, so it’s not a complete mess.

Python 3.4 is already EOL. We should not support it for the complete build system, but I can see that the install-buildtools script should ideally still run on python34. The whole point is to make it easier to install the pre-built buildtools.

Debian-8 is EOL in June 2020.

Quoted 62 lines

scripts/install-buildtools if not bitbakepath: · line 63

Quoted 1 line
scripts/install-buildtools +0 -0
 DEFAULT_BASE_URL: str = '

http://downloads.yoctoproject.org/releases/yocto'

Quoted 19 lines
scripts/install-buildtools +2 -0
 DEFAULT_RELEASE: str = 'yocto-3.1_M2'
 DEFAULT_INSTALLER_VERSION: str = '3.0+snapshot'
@@ -66,6 +67,7 @@ DEFAULT_BUILDDATE: str = "20200122"
 
 
 def main():
+    global DEFAULT_INSTALL_DIR
     global DEFAULT_BASE_URL
     global DEFAULT_RELEASE
     global DEFAULT_INSTALLER_VERSION
@@ -73,6 +75,7 @@ def main():
     filename: str = ""
     release: str = ""
     buildtools_url: str = ""
+    install_dir: str = ""
 
     parser = argparse.ArgumentParser(
         description="Buildtools installation helper",

scripts/install-buildtools def main(): · line 94

Quoted 5 lines
scripts/install-buildtools +1 -0
@@ -87,6 +90,7 @@ def main():
                              '(optional)\nRequires --url',
                         action='store')
     parser.add_argument('-d', '--directory',
+                        default=DEFAULT_INSTALL_DIR,
                         help='directory where buildtools SDK will be

installed (optional)',

scripts/install-buildtools def main(): · line 226

Quoted 11 lines
scripts/install-buildtools +0 -1
                         action='store')
     parser.add_argument('-r', '--release',
@@ -216,12 +220,12 @@ def main():
         st = os.stat(tmpbuildtools)
         os.chmod(tmpbuildtools, st.st_mode | stat.S_IEXEC)
         logger.debug(os.stat(tmpbuildtools))
-        install_dir = "/opt/poky/%s" % args.installer_version
         if args.directory:
             install_dir = args.directory
             ret = subprocess.call("%s -d %s -y" %
                                   (tmpbuildtools, install_dir),

shell=True)

scripts/install-buildtools def main(): · line 241

Quoted 10 lines
scripts/install-buildtools +1 -1
         else:
+            install_dir = "/opt/poky/%s" % args.installer_version
             ret = subprocess.call("%s -y" % tmpbuildtools, shell=True)
         if ret != 0:
             logger.error("Could not run buildtools installer")
@@ -238,7 +242,8 @@ def main():
             tool = 'gcc'
         else:
             tool = 'tar'
-        proc = subprocess.run("source

%s/environment-setup-x86_64-pokysdk-linux && which %s" %

scripts/install-buildtools def main(): · line 246

Quoted 2 lines
scripts/install-buildtools +2 -0
+        logger.debug("install_dir: %s" % install_dir)
+        proc = subprocess.run(".

%s/environment-setup-x86_64-pokysdk-linux && which %s" %

Quoted 7 lines
scripts/install-buildtools +0 -1
                               (install_dir, tool),
                               shell=True, stdout=subprocess.PIPE)
         which_tool = proc.stdout.decode("utf-8")
--

2.24.0

A message that is not in the archive (sent off-list or not yet received): [email protected]

#9 lore raw
Quoted 78 lines · Tim Orling via lists.openembedded.org
Quoted 63 lines · Andre McCurdy
Quoted 38 lines · Tim Orling

* Install directory defaults to scripts/../buildtools e.g. --directory is set by default
This avoids the user having to type in their sudo password to install in /opt/poky/<installer-version>

* Use "." rather than "source" for sourcing the environment script as not all distros (e.g. Debian) have "source" by default.

* Add buildtools/ to .gitignore

* Fix typos in example usage (--install-version -> --installer-version)

[YOCTO #13832]

---
 .gitignore                 |  1 +
 scripts/install-buildtools | 13 +++++++++----
 2 files changed, 10 insertions(+), 4 deletions(-)
.gitignore +1 -0
index d0e6b2fb89..b66d371aac 100644
@@ -9,6 +9,7 @@ pstage/
 scripts/oe-git-proxy-socks
 sources/
 meta-*/
+buildtools/
 !meta-skeleton
 !meta-selftest
 hob-image-*.bb
scripts/install-buildtools +0 -0
index 0947e9c4d6..49cab1345a 100755

scripts/install-buildtools line 18

Quoted 2 lines
scripts/install-buildtools +0 -0
@@ -17,7 +17,7 @@
 #        $ install-buildtools \
 #          --base-url http://downloads.yoctoproject.org/releases/yocto

\

scripts/install-buildtools line 30

Quoted 9 lines
scripts/install-buildtools +1 -1
 #          --release yocto-3.1_M2 \
-#          --install-version 3.0+snapshot
+#          --installer-version 3.0+snapshot
 #          --build-date 202000122
 #
 #  Example usage (standard buildtools from release):
@@ -29,7 +29,7 @@
 #        $ install-buildtools --without-extended-buildtools \
 #          --base-url http://downloads.yoctoproject.org/releases/yocto

\

Quoted 10 lines
scripts/install-buildtools +1 -1
 #          --release yocto-3.0.2 \
-#          --install-version 3.0.2
+#          --installer-version 3.0.2
 #
 
 import argparse
@@ -59,6 +59,7 @@ if not bitbakepath:
 PROGNAME = 'install-buildtools'
 logger = scriptutils.logger_create(PROGNAME, stream=sys.stdout)
 
Quoted 7 lines

scripts/install-buildtools if not bitbakepath: · line 62

Quoted 1 line
scripts/install-buildtools +1 -0
+DEFAULT_INSTALL_DIR: str =

os.path.join(os.path.split(scripts_path)[0],'buildtools')

It looks like there are some assumptions about the minimum version of python3 on the host in order to even be able to run install-buildtools. Should that be checked explicitly?

To be honest, everything was written on CentOS-7, where ‘python3’ is now python36.

Quoted 9 lines

The remaining supported holdout for 3.4 is Debian-8 (Jessie). Which also happens to have tar 1.27. At least it has git 2.1.4, so it’s not a complete mess.

Python 3.4 is already EOL. We should not support it for the complete build system, but I can see that the install-buildtools script should ideally still run on python34. The whole point is to make it easier to install the pre-built buildtools.

Python 3.5 is now required in bitbake/lib/bb/__init__.py Which is needed for bb.utils.md5sum_file and sha256_file to check the checksum of the download. I’d rather not have to rewrite those functions.

This means Debian-8 is a conundrum.

Quoted 66 lines

Debian-8 is EOL in June 2020.

Quoted 62 lines

scripts/install-buildtools if not bitbakepath: · line 63

Quoted 1 line
scripts/install-buildtools +0 -0
 DEFAULT_BASE_URL: str = '

http://downloads.yoctoproject.org/releases/yocto'

Quoted 19 lines
scripts/install-buildtools +2 -0
 DEFAULT_RELEASE: str = 'yocto-3.1_M2'
 DEFAULT_INSTALLER_VERSION: str = '3.0+snapshot'
@@ -66,6 +67,7 @@ DEFAULT_BUILDDATE: str = "20200122"
 
 
 def main():
+    global DEFAULT_INSTALL_DIR
     global DEFAULT_BASE_URL
     global DEFAULT_RELEASE
     global DEFAULT_INSTALLER_VERSION
@@ -73,6 +75,7 @@ def main():
     filename: str = ""
     release: str = ""
     buildtools_url: str = ""
+    install_dir: str = ""
 
     parser = argparse.ArgumentParser(
         description="Buildtools installation helper",

scripts/install-buildtools def main(): · line 94

Quoted 5 lines
scripts/install-buildtools +1 -0
@@ -87,6 +90,7 @@ def main():
                              '(optional)\nRequires --url',
                         action='store')
     parser.add_argument('-d', '--directory',
+                        default=DEFAULT_INSTALL_DIR,
                         help='directory where buildtools SDK will be

installed (optional)',

scripts/install-buildtools def main(): · line 226

Quoted 11 lines
scripts/install-buildtools +0 -1
                         action='store')
     parser.add_argument('-r', '--release',
@@ -216,12 +220,12 @@ def main():
         st = os.stat(tmpbuildtools)
         os.chmod(tmpbuildtools, st.st_mode | stat.S_IEXEC)
         logger.debug(os.stat(tmpbuildtools))
-        install_dir = "/opt/poky/%s" % args.installer_version
         if args.directory:
             install_dir = args.directory
             ret = subprocess.call("%s -d %s -y" %
                                   (tmpbuildtools, install_dir),

shell=True)

scripts/install-buildtools def main(): · line 241

Quoted 10 lines
scripts/install-buildtools +1 -1
         else:
+            install_dir = "/opt/poky/%s" % args.installer_version
             ret = subprocess.call("%s -y" % tmpbuildtools, shell=True)
         if ret != 0:
             logger.error("Could not run buildtools installer")
@@ -238,7 +242,8 @@ def main():
             tool = 'gcc'
         else:
             tool = 'tar'
-        proc = subprocess.run("source

%s/environment-setup-x86_64-pokysdk-linux && which %s" %

scripts/install-buildtools def main(): · line 246

Quoted 2 lines
scripts/install-buildtools +2 -0
+        logger.debug("install_dir: %s" % install_dir)
+        proc = subprocess.run(".

%s/environment-setup-x86_64-pokysdk-linux && which %s" %

Quoted 7 lines
scripts/install-buildtools +0 -1
                               (install_dir, tool),
                               shell=True, stdout=subprocess.PIPE)
         which_tool = proc.stdout.decode("utf-8")
--

2.24.0

#10 lore raw
Quoted 71 lines · Tim Orling
Quoted 70 lines · Tim Orling via lists.openembedded.org
Quoted 67 lines · Andre McCurdy
Quoted 56 lines · Tim Orling

* Install directory defaults to scripts/../buildtools e.g. --directory is set by default
This avoids the user having to type in their sudo password to install in /opt/poky/<installer-version>

* Use "." rather than "source" for sourcing the environment script as not all distros (e.g. Debian) have "source" by default.

* Add buildtools/ to .gitignore

* Fix typos in example usage (--install-version -> --installer-version)

[YOCTO #13832]

---
 .gitignore                 |  1 +
 scripts/install-buildtools | 13 +++++++++----
 2 files changed, 10 insertions(+), 4 deletions(-)
.gitignore +1 -0
index d0e6b2fb89..b66d371aac 100644
@@ -9,6 +9,7 @@ pstage/
 scripts/oe-git-proxy-socks
 sources/
 meta-*/
+buildtools/
 !meta-skeleton
 !meta-selftest
 hob-image-*.bb
scripts/install-buildtools +2 -2
index 0947e9c4d6..49cab1345a 100755
@@ -17,7 +17,7 @@
 #        $ install-buildtools \
 #          --base-url http://downloads.yoctoproject.org/releases/yocto \
 #          --release yocto-3.1_M2 \
-#          --install-version 3.0+snapshot
+#          --installer-version 3.0+snapshot
 #          --build-date 202000122
 #
 #  Example usage (standard buildtools from release):
@@ -29,7 +29,7 @@
 #        $ install-buildtools --without-extended-buildtools \
 #          --base-url http://downloads.yoctoproject.org/releases/yocto \
 #          --release yocto-3.0.2 \
-#          --install-version 3.0.2
+#          --installer-version 3.0.2
 #
 
 import argparse

scripts/install-buildtools if not bitbakepath: · line 62

Quoted 4 lines
scripts/install-buildtools +1 -0
@@ -59,6 +59,7 @@ if not bitbakepath:
 PROGNAME = 'install-buildtools'
 logger = scriptutils.logger_create(PROGNAME, stream=sys.stdout)
 
+DEFAULT_INSTALL_DIR: str = os.path.join(os.path.split(scripts_path)[0],'buildtools')

It looks like there are some assumptions about the minimum version of python3 on the host in order to even be able to run install-buildtools. Should that be checked explicitly?

Quoted 8 lines
Quoted 5 lines

To be honest, everything was written on CentOS-7, where ‘python3’ is now python36.

The remaining supported holdout for 3.4 is Debian-8 (Jessie). Which also happens to have tar 1.27. At least it has git 2.1.4, so it’s not a complete mess.

Python 3.4 is already EOL. We should not support it for the complete build system, but I can see that the install-buildtools script should ideally still run on python34. The whole point is to make it easier to install the pre-built buildtools.

Python 3.5 is now required in bitbake/lib/bb/__init__.py
Which is needed for bb.utils.md5sum_file and sha256_file to check the checksum of the download. I’d rather not have to rewrite those functions.

Looks like copy and pasting ~30 lines? Maybe not even much of a net increase in code size if you can remove the existing code to check that bitbake libs are available...

def _hasher(method, filename):
import mmap

with open(filename, "rb") as f:
try:
with mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) as mm:
for chunk in iter(lambda: mm.read(8192), b''):
method.update(chunk)
except ValueError:
# You can't mmap() an empty file so silence this exception pass
return method.hexdigest()

def md5_file(filename):
"""
Return the hex string representation of the MD5 checksum of filename. """
import hashlib
return _hasher(hashlib.md5(), filename)

def sha256_file(filename):
"""
Return the hex string representation of the 256-bit SHA checksum of filename.
"""
import hashlib
return _hasher(hashlib.sha256(), filename)

Quoted 60 lines
Quoted 60 lines

Debian-8 is EOL in June 2020.

Quoted 57 lines
Quoted 56 lines
scripts/install-buildtools +6 -3
 DEFAULT_BASE_URL: str = 'http://downloads.yoctoproject.org/releases/yocto'
 DEFAULT_RELEASE: str = 'yocto-3.1_M2'
 DEFAULT_INSTALLER_VERSION: str = '3.0+snapshot'
@@ -66,6 +67,7 @@ DEFAULT_BUILDDATE: str = "20200122"
 
 
 def main():
+    global DEFAULT_INSTALL_DIR
     global DEFAULT_BASE_URL
     global DEFAULT_RELEASE
     global DEFAULT_INSTALLER_VERSION
@@ -73,6 +75,7 @@ def main():
     filename: str = ""
     release: str = ""
     buildtools_url: str = ""
+    install_dir: str = ""
 
     parser = argparse.ArgumentParser(
         description="Buildtools installation helper",
@@ -87,6 +90,7 @@ def main():
                              '(optional)\nRequires --url',
                         action='store')
     parser.add_argument('-d', '--directory',
+                        default=DEFAULT_INSTALL_DIR,
                         help='directory where buildtools SDK will be installed (optional)',
                         action='store')
     parser.add_argument('-r', '--release',
@@ -216,12 +220,12 @@ def main():
         st = os.stat(tmpbuildtools)
         os.chmod(tmpbuildtools, st.st_mode | stat.S_IEXEC)
         logger.debug(os.stat(tmpbuildtools))
-        install_dir = "/opt/poky/%s" % args.installer_version
         if args.directory:
             install_dir = args.directory
             ret = subprocess.call("%s -d %s -y" %
                                   (tmpbuildtools, install_dir), shell=True)
         else:
+            install_dir = "/opt/poky/%s" % args.installer_version
             ret = subprocess.call("%s -y" % tmpbuildtools, shell=True)
         if ret != 0:
             logger.error("Could not run buildtools installer")
@@ -238,7 +242,8 @@ def main():
             tool = 'gcc'
         else:
             tool = 'tar'
-        proc = subprocess.run("source %s/environment-setup-x86_64-pokysdk-linux && which %s" %
+        logger.debug("install_dir: %s" % install_dir)
+        proc = subprocess.run(". %s/environment-setup-x86_64-pokysdk-linux && which %s" %
                               (install_dir, tool),
                               shell=True, stdout=subprocess.PIPE)
         which_tool = proc.stdout.decode("utf-8")
--

2.24.0

#11 lore raw
Quoted 55 lines · Andre McCurdy
Quoted 22 lines · Tim Orling
Quoted 10 lines · Tim Orling via lists.openembedded.org

On Wed, Apr 1, 2020 at 1:09 PM Andre McCurdy <[email protected]

Quoted 5 lines

wrote:
It looks like there are some assumptions about the minimum version of
python3 on the host in order to even be able to run install-buildtools. Should that be checked explicitly?

To be honest, everything was written on CentOS-7, where ‘python3’ is now python36.

Quoted 8 lines

The remaining supported holdout for 3.4 is Debian-8 (Jessie). Which also happens to have tar 1.27. At least it has git 2.1.4, so it’s not a complete mess.

Python 3.4 is already EOL. We should not support it for the complete build system, but I can see that the install-buildtools script should ideally still run on python34. The whole point is to make it easier to install the pre-built buildtools.

Quoted 4 lines

Python 3.5 is now required in bitbake/lib/bb/__init__.py Which is needed for bb.utils.md5sum_file and sha256_file to check the checksum of the download. I’d rather not have to rewrite those functions.

Looks like copy and pasting ~30 lines? Maybe not even much of a net increase in code size if you can remove the existing code to check that bitbake libs are available...

def _hasher(method, filename):
import mmap

with open(filename, "rb") as f:
try:
with mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) as
mm:
for chunk in iter(lambda: mm.read(8192), b''):
method.update(chunk)
except ValueError:
# You can't mmap() an empty file so silence this
exception
pass
return method.hexdigest()

def md5_file(filename):
"""
Return the hex string representation of the MD5 checksum of
filename.
"""
import hashlib
return _hasher(hashlib.md5(), filename)

Quoted 8 lines

def sha256_file(filename):
"""
Return the hex string representation of the 256-bit SHA checksum
of
filename.
"""
import hashlib
return _hasher(hashlib.sha256(), filename)

Can you send a patch against master-next please? The buildtools installer is one place we probably do want to support older pythons and it shouldn't be too onnerous.

Cheers,

Richard

author
#12 lore raw
Quoted 69 lines · Richard Purdie
Quoted 55 lines · Andre McCurdy
Quoted 22 lines · Tim Orling
Quoted 10 lines · Tim Orling via lists.openembedded.org

On Wed, Apr 1, 2020 at 1:09 PM Andre McCurdy <[email protected]

Quoted 5 lines

wrote:
It looks like there are some assumptions about the minimum version of
python3 on the host in order to even be able to run install-buildtools. Should that be checked explicitly?

To be honest, everything was written on CentOS-7, where ‘python3’ is now python36.

Quoted 8 lines

The remaining supported holdout for 3.4 is Debian-8 (Jessie). Which also happens to have tar 1.27. At least it has git 2.1.4, so it’s not a complete mess.

Python 3.4 is already EOL. We should not support it for the complete build system, but I can see that the install-buildtools script should ideally still run on python34. The whole point is to make it easier to install the pre-built buildtools.

Quoted 4 lines

Python 3.5 is now required in bitbake/lib/bb/__init__.py Which is needed for bb.utils.md5sum_file and sha256_file to check the checksum of the download. I’d rather not have to rewrite those functions.

Looks like copy and pasting ~30 lines? Maybe not even much of a net increase in code size if you can remove the existing code to check that bitbake libs are available...

def _hasher(method, filename):
import mmap

with open(filename, "rb") as f:
try:
with mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) as
mm:
for chunk in iter(lambda: mm.read(8192), b''):
method.update(chunk)
except ValueError:
# You can't mmap() an empty file so silence this
exception
pass
return method.hexdigest()

def md5_file(filename):
"""
Return the hex string representation of the MD5 checksum of
filename.
"""
import hashlib
return _hasher(hashlib.md5(), filename)

Quoted 8 lines

def sha256_file(filename):
"""
Return the hex string representation of the 256-bit SHA checksum
of
filename.
"""
import hashlib
return _hasher(hashlib.sha256(), filename)

Quoted 4 lines

Can you send a patch against master-next please? The buildtools installer is one place we probably do want to support older pythons and it shouldn't be too onnerous.

Submitted. The check for whether the install worked or not was the hardest part. After many iterations, I just could not get subprocess.Popen to work as expected on python 3.4.
https://patchwork.openembedded.org/patch/171593/ <https://patchwork.openembedded.org/patch/171593/>

Thank you Andre for the testing and input. It takes a village to make a good release.

Quoted 5 lines

Cheers,

Richard