Make oe.lsb.distro_identifier() more consistent
v2 cover Make oe.lsb.distro_identifier() more consistent
The oe.lsb.distro_identifier() method call will return different identification information depending on the source which is found to provide that information.
This series attempts to address this in two ways:
1) preferring os-release(5) as the source of distribution identification. this increasingly common standard mechanism is available on each of the build host distributions we commonly test on.
2) converting the distribution identifier to lower case before including it in the distro_identifier return value. This ensures that, for most of the tested distros, the identifier returned via the LSB code paths matches that returned by the os-release code paths.
Changes since v1:
* improve release_dict_osr() in patch 3 to handle empty lines and values with surrounding quotation marks in /etc/os-release, as used in CentOS
The following changes since commit 9303d8055c45a0f6af295d70a6f6a8b9d8d8a7c9:
devtool: add "rename" subcommand (2016-11-07 11:04:17 +0000)
are available in the git repository at:
git://git.openembedded.org/openembedded-core-contrib joshuagl/liboe
http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=joshuagl/liboe
Joshua Lock (4):
lib/oe/path: remove duplicate import
lib/oe/lsb: make the release dict keys consistent regardless of source lib/oe/lsb: prefer /etc/os-release for distribution data lib/oe/lsb: attempt to ensure consistent distro id regardless of
source
meta/lib/oe/lsb.py | 73 +++++++++++++++++++++++++++++++++++------------------ meta/lib/oe/path.py | 1 - 2 files changed, 49 insertions(+), 25 deletions(-)
Signature · 1 line
2.7.4
PATCH v2 1/4 lib/oe/path: remove duplicate import
There's no need to import glob inside copyhardlinktree() as it's already imported for the entire path module.
--- meta/lib/oe/path.py | 1 - 1 file changed, 1 deletion(-)
meta/lib/oe/path.py +0 -1
@@ -81,7 +81,6 @@ def copyhardlinktree(src, dst): subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT) source = '' if os.path.isdir(src): - import glob if len(glob.glob('%s/.??*' % src)) > 0: source = '%s/.??* ' % src source = source + '%s/*' % src
Signature · 1 line
2.7.4
PATCH v2 2/4 lib/oe/lsb: make the release dict keys consistent regardless of source
In reply to Joshua LockRather than have the distro_identifier method look for different keys in the dict depending on the source ensure that each function for retrieving release data uses the same key names in the returned dict.
--- meta/lib/oe/lsb.py | 36 +++++++++++++++++++++--------------- 1 file changed, 21 insertions(+), 15 deletions(-)
meta/lib/oe/lsb.py +21 -15
@@ -1,5 +1,5 @@ -def release_dict(): - """Return the output of lsb_release -ir as a dictionary""" +def release_dict_lsb(): + """ Return the output of lsb_release -ir as a dictionary """ from subprocess import PIPE try:
@@ -7,19 +7,28 @@ def release_dict(): except bb.process.CmdError as exc: return None + lsb_map = { 'Distributor ID': 'DISTRIB_ID', + 'Release': 'DISTRIB_RELEASE'} + lsb_keys = lsb_map.keys() + data = {} for line in output.splitlines(): - if line.startswith("-e"): line = line[3:] + if line.startswith("-e"): + line = line[3:] try: key, value = line.split(":\t", 1) except ValueError: continue - else: - data[key] = value + if key in lsb_keys: + data[lsb_map[key]] = value + + if len(data.keys()) != 2: + return None + return data def release_dict_file(): - """ Try to gather LSB release information manually when lsb_release tool is unavailable """ + """ Try to gather release information manually when other methods fail """ data = None try: if os.path.exists('/etc/lsb-release'):
@@ -64,15 +73,12 @@ def distro_identifier(adjust_hook=None): import re - lsb_data = release_dict() - if lsb_data: - distro_id, release = lsb_data['Distributor ID'], lsb_data['Release'] - else: - lsb_data_file = release_dict_file() - if lsb_data_file: - distro_id, release = lsb_data_file['DISTRIB_ID'], lsb_data_file.get('DISTRIB_RELEASE', None) - else: - distro_id, release = None, None + distro_data = release_dict_lsb() + if not distro_data: + distro_data = release_dict_file() + + distro_id = distro_data['DISTRIB_ID'] + release = distro_data['DISTRIB_RELEASE'] if adjust_hook: distro_id, release = adjust_hook(distro_id, release)
Signature · 1 line
2.7.4
PATCH v2 3/4 lib/oe/lsb: prefer /etc/os-release for distribution data
In reply to Joshua Lockos-release(5) is an increasingly standard source of operating system identification and more likely to be present on modern OS deployments, i.e. many container variants of common distros include os-release and not the lsb_release tool.
Therefore we should favour parsing /etc/os-release in distro_identifier(), try lsb_release when that fails and finally fall back on various distro specific sources of OS identification.
--- meta/lib/oe/lsb.py | 37 ++++++++++++++++++++++++++++--------- 1 file changed, 28 insertions(+), 9 deletions(-)
meta/lib/oe/lsb.py +28 -9
@@ -1,3 +1,25 @@ +def release_dict_osr(): + """ Populate a dict with pertinent values from /etc/os-release """ + if not os.path.exists('/etc/os-release'): + return None + + data = {} + with open('/etc/os-release') as f: + for line in f: + try: + key, val = line.rstrip().split('=', 1) + except ValueError: + continue + if key == 'NAME': + data['DISTRIB_ID'] = val.strip('"') + if key == 'VERSION_ID': + data['DISTRIB_RELEASE'] = val.strip('"') + + if len(data.keys()) != 2: + return None + + return data + def release_dict_lsb(): """ Return the output of lsb_release -ir as a dictionary """ from subprocess import PIPE
@@ -46,14 +68,6 @@ def release_dict_file(): if match: data['DISTRIB_ID'] = match.group(1) data['DISTRIB_RELEASE'] = match.group(2) - elif os.path.exists('/etc/os-release'): - data = {} - with open('/etc/os-release') as f: - for line in f: - if line.startswith('NAME='): - data['DISTRIB_ID'] = line[5:].rstrip().strip('"') - if line.startswith('VERSION_ID='): - data['DISTRIB_RELEASE'] = line[11:].rstrip().strip('"') elif os.path.exists('/etc/SuSE-release'): data = {} data['DISTRIB_ID'] = 'SUSE LINUX'
@@ -73,7 +87,12 @@ def distro_identifier(adjust_hook=None): import re - distro_data = release_dict_lsb() + # Try /etc/os-release first, then the output of `lsb_release -ir` and + # finally fall back on parsing various release files in order to determine + # host distro name and version. + distro_data = release_dict_osr() + if not distro_data: + distro_data = release_dict_lsb() if not distro_data: distro_data = release_dict_file()
Signature · 1 line
2.7.4
PATCH v2 4/4 lib/oe/lsb: attempt to ensure consistent distro id regardless of source
In reply to Joshua LockThe LSB Distributor ID and os-release NAME differ for most of the distributions tested by the Yocto Project (CentOS, Debian, Fedora, openSUSE and Ubuntu) however for all but openSUSE the os-release ID matches the LSB Distributor ID when both are lowered before comparison.
Therefore, in order to improve the consistency of identification of a distribution, switch to using the os-release ID and converting the ID value to lowercase.
Table showing comparison of LSB Distributor ID to os-release fields NAME and ID for current Yocto Project supported host distributions:
Distribution | Version | Distributor ID | NAME | ID |
-------------------------------------------------------------------------
CentOS | 7 | CentOS | CentOS Linux | centos | Debian | 8 | Debian | Debian GNU/Linux | debian | Fedora | 23 | Fedora | Fedora | fedora | Fedora | 24 | Fedora | Fedora | fedora | openSUSE | 13.2 | openSUSE project | openSUSE | opensuse | openSUSE | 42.1 | SUSE LINUX | openSUSE Leap | opensuse | Ubuntu | 14.04 | Ubuntu | Ubuntu | ubuntu | Ubuntu | 16.04 | Ubuntu | Ubuntu | ubuntu |
[YOCTO #10591]
--- meta/lib/oe/lsb.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
meta/lib/oe/lsb.py +2 -2
@@ -10,7 +10,7 @@ def release_dict_osr(): key, val = line.rstrip().split('=', 1) except ValueError: continue - if key == 'NAME': + if key == 'ID': data['DISTRIB_ID'] = val.strip('"') if key == 'VERSION_ID': data['DISTRIB_RELEASE'] = val.strip('"')
@@ -107,7 +107,7 @@ def distro_identifier(adjust_hook=None): distro_id = re.sub(r'\W', '', distro_id) if release: - id_str = '{0}-{1}'.format(distro_id, release) + id_str = '{0}-{1}'.format(distro_id.lower(), release) else: id_str = distro_id return id_str.replace(' ','-').replace('/','-')
Signature · 1 line
2.7.4