package: moving field data process before variable process in process_pkgconfig
PATCH v2 1/1 package: moving field data process before variable process in process_pkgconfig
From: Xiangyu Chen <[email protected]>
Currently, the latest version abseil-cpp contains a new library named "absl_log_internal_format", it's basic package config(.pc file) as below:
prefix=/usr
exec_prefix=${prefix}
......
Requires: absl_config = 20230125, absl_core_headers = 20230125, absl_log_internal_append_truncated = 20230125, absl_log_internal_config = 20230125, absl_log_internal_globals = 20230125, absl_log_severity = 20230125, absl_strings = 20230125, absl_str_format = 20230125, absl_time = 20230125, absl_span = 20230125
......
Normally, the process_pkgconfig() would process variable data before field data in a .pc file, but in the absl_log_internal_format, the field data in "Requires" section contains "xxxx = xxxx" format, the process_pkgconfig() treats them as normal variable and using the setVar() in bitbake's data_smart.py try to process. The absl_log_internal_format field data contains "_append_", this hit the setVar() checking and finally bitbake stop building and reporting an error as below:
"Variable xxx contains an operation using the old override syntax. Please convert this layer/metadata before attempting to use with a newer bitbake."
This patch move the field data process before variable process to avoid the process_pkgconfig() treat the field data as variable.
---
Changes:
V1 -> V2: changing the package.py instead of bitbake's data_smart.py v1 thread please refer:https://lists.openembedded.org/g/bitbake-devel/message/14525
--- meta/lib/oe/package.py | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)
meta/lib/oe/package.py +6 -6
@@ -1823,18 +1823,18 @@ def process_pkgconfig(pkgfiles, d): with open(file, 'r') as f: lines = f.readlines() for l in lines: - m = var_re.match(l) - if m: - name = m.group(1) - val = m.group(2) - pd.setVar(name, pd.expand(val)) - continue m = field_re.match(l) if m: hdr = m.group(1) exp = pd.expand(m.group(2)) if hdr == 'Requires': pkgconfig_needed[pkg] += exp.replace(',', ' ').split() + continue + m = var_re.match(l) + if m: + name = m.group(1) + val = m.group(2) + pd.setVar(name, pd.expand(val)) for pkg in packages.split(): pkgs_file = os.path.join(shlibswork_dir, pkg + ".pclist")
Signature · 1 line
2.34.1
Quoted 26 lines · Xiangyu Chen
From: Xiangyu Chen <[email protected]>
Currently, the latest version abseil-cpp contains a new library named "
absl_log_internal_format", it's basic package config(.pc file) as below:prefix=/usr
exec_prefix=${prefix}......
Requires:
absl_config= 20230125,absl_core_headers= 20230125,absl_log_internal_append_truncated= 20230125,absl_log_internal_config= 20230125,absl_log_internal_globals= 20230125,absl_log_severity= 20230125,absl_strings= 20230125,absl_str_format= 20230125,absl_time= 20230125,absl_span= 20230125
......Normally, the
process_pkgconfig()would process variable data before field data in a .pc file, but in theabsl_log_internal_format, the field data in "Requires" section contains "xxxx = xxxx" format, theprocess_pkgconfig()treats them as normal variable and using thesetVar()in bitbake'sdata_smart.pytry to process. Theabsl_log_internal_formatfield data contains "_append_", this hit thesetVar()checking and finally bitbake stop building and reporting an error as below:"Variable xxx contains an operation using the old override syntax. Please convert this layer/metadata before attempting to use with a newer bitbake."
This patch move the field data process before variable process to avoid the
process_pkgconfig()treat the field data as variable.
Quoted 6 lines
Signed-off-by Xiangyu Chen <[email protected]>---
Changes:
V1 -> V2: changing thepackage.pyinstead of bitbake'sdata_smart.pyv1 thread please refer:https://lists.openembedded.org/g/bitbake-devel/message/14525
This looks like a much better fix than the workaround in v1, thanks!
Cheers,
Richard
Quoted 56 lines · Xiangyu Chen
From: Xiangyu Chen <[email protected]>
Currently, the latest version abseil-cpp contains a new library named "
absl_log_internal_format", it's basic package config(.pc file) as below:prefix=/usr
exec_prefix=${prefix}......
Requires:
absl_config= 20230125,absl_core_headers= 20230125,absl_log_internal_append_truncated= 20230125,absl_log_internal_config= 20230125,absl_log_internal_globals= 20230125,absl_log_severity= 20230125,absl_strings= 20230125,absl_str_format= 20230125,absl_time= 20230125,absl_span= 20230125
......Normally, the
process_pkgconfig()would process variable data before field data in a .pc file, but in theabsl_log_internal_format, the field data in "Requires" section contains "xxxx = xxxx" format, theprocess_pkgconfig()treats them as normal variable and using thesetVar()in bitbake'sdata_smart.pytry to process. Theabsl_log_internal_formatfield data contains "_append_", this hit thesetVar()checking and finally bitbake stop building and reporting an error as below:"Variable xxx contains an operation using the old override syntax. Please convert this layer/metadata before attempting to use with a newer bitbake."
This patch move the field data process before variable process to avoid the
process_pkgconfig()treat the field data as variable.Signed-off-by Xiangyu Chen <[email protected]>---
Changes:
V1 -> V2: changing thepackage.pyinstead of bitbake'sdata_smart.pyv1 thread please refer:https://lists.openembedded.org/g/bitbake-devel/message/14525--- meta/lib/oe/package.py | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)meta/lib/oe/package.py +0
-6@@ -1823,18 +1823,18 @@ def process_pkgconfig(pkgfiles, d): with open(file, 'r') as f: lines = f.readlines() for l in lines: - m = var_re.match(l) - if m: - name = m.group(1) - val = m.group(2) - pd.setVar(name, pd.expand(val)) - continue m = field_re.match(l) if m: hdr = m.group(1) exp = pd.expand(m.group(2)) if hdr == 'Requires':
meta/lib/oe/package.py def process_pkgconfig(pkgfiles, d): · line 1840
Quoted 10 lines
meta/lib/oe/package.py +6
-0pkgconfig_needed[pkg] += exp.replace(',', ' ').split() + continue + m = var_re.match(l) + if m: + name = m.group(1) + val = m.group(2) + pd.setVar(name, pd.expand(val)) for pkg in packages.split(): pkgs_file = os.path.join(shlibswork_dir, pkg + ".pclist")
It is worth noting this fix actually resolved a number of latent bugs too. This breakage in reproducibility was as a result of this change:
http://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20230325-0k1j3_4w/packages/diff-html/
which was because the output changed but the sstate hashes didn't. That is because package.py code isn't part of the sstate hashes (currently).
This isn't a fault of this patch. I've a patch in progress to bump various versions to make sure this doesn't break anything further and it does raise a question on whether we should checksum the python functions and track that due to the adverse impact an issue like this can have.
The issue here was missing dependencies, which were missing from -dev packages when there was a version constraint present. I wanted to highlight why fixing things at the root cause is good! If we'd hacked this in bitbake, the other issues would have remained unseen.
Cheers,
Richard