Skip to contentSkip to conversation

package: moving field data process before variable process in process_pkgconfig

PATCH In review Xiangyu Chen 2 participants 3 messages Mar 24, 2023 – Mar 26, 2023

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 package: moving field data process before variable process in process_pkgconfig

From: Xiangyu Chen <[email protected]>

Currently, the latest version abseil-cpp contains a new library named "absl_log_internal_format", it's basic package config(.pc file) as below:

prefix=/usr
exec_prefix=${prefix}

......

Requires: absl_config = 20230125, absl_core_headers = 20230125, absl_log_internal_append_truncated = 20230125, absl_log_internal_config = 20230125, absl_log_internal_globals = 20230125, absl_log_severity = 20230125, absl_strings = 20230125, absl_str_format = 20230125, absl_time = 20230125, absl_span = 20230125
......

Normally, the process_pkgconfig() would process variable data before field data in a .pc file, but in the absl_log_internal_format, the field data in "Requires" section contains "xxxx = xxxx" format, the process_pkgconfig() treats them as normal variable and using the setVar() in bitbake's data_smart.py try to process. The absl_log_internal_format field data contains "_append_", this hit the setVar() checking and finally bitbake stop building and reporting an error as below:

"Variable xxx contains an operation using the old override syntax. Please convert this layer/metadata before attempting to use with a newer bitbake."

This patch move the field data process before variable process to avoid the process_pkgconfig() treat the field data as variable.

---

Changes:
V1 -> V2: changing the package.py instead of bitbake's data_smart.py v1 thread please refer:https://lists.openembedded.org/g/bitbake-devel/message/14525

---
 meta/lib/oe/package.py | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)
meta/lib/oe/package.py +6 -6
index c9eb75d852..7a6b31957a 100644
@@ -1823,18 +1823,18 @@ def process_pkgconfig(pkgfiles, d):
                     with open(file, 'r') as f:
                         lines = f.readlines()
                     for l in lines:
-                        m = var_re.match(l)
-                        if m:
-                            name = m.group(1)
-                            val = m.group(2)
-                            pd.setVar(name, pd.expand(val))
-                            continue
                         m = field_re.match(l)
                         if m:
                             hdr = m.group(1)
                             exp = pd.expand(m.group(2))
                             if hdr == 'Requires':
                                 pkgconfig_needed[pkg] += exp.replace(',', ' ').split()
+                                continue
+                        m = var_re.match(l)
+                        if m:
+                            name = m.group(1)
+                            val = m.group(2)
+                            pd.setVar(name, pd.expand(val))
 
     for pkg in packages.split():
         pkgs_file = os.path.join(shlibswork_dir, pkg + ".pclist")
Signature · 1 line
2.34.1
#1 lore raw
Quoted 26 lines · Xiangyu Chen

From: Xiangyu Chen <[email protected]>

Currently, the latest version abseil-cpp contains a new library named "absl_log_internal_format", it's basic package config(.pc file) as below:

prefix=/usr
exec_prefix=${prefix}

......

Requires: absl_config = 20230125, absl_core_headers = 20230125, absl_log_internal_append_truncated = 20230125, absl_log_internal_config = 20230125, absl_log_internal_globals = 20230125, absl_log_severity = 20230125, absl_strings = 20230125, absl_str_format = 20230125, absl_time = 20230125, absl_span = 20230125
......

Normally, the process_pkgconfig() would process variable data before field data in a .pc file, but in the absl_log_internal_format, the field data in "Requires" section contains "xxxx = xxxx" format, the process_pkgconfig() treats them as normal variable and using the setVar() in bitbake's data_smart.py try to process. The absl_log_internal_format field data contains "_append_", this hit the setVar() checking and finally bitbake stop building and reporting an error as below:

"Variable xxx contains an operation using the old override syntax. Please convert this layer/metadata before attempting to use with a newer bitbake."

This patch move the field data process before variable process to avoid the process_pkgconfig() treat the field data as variable.

Quoted 6 lines

---

Changes:
V1 -> V2: changing the package.py instead of bitbake's data_smart.py v1 thread please refer:https://lists.openembedded.org/g/bitbake-devel/message/14525

This looks like a much better fix than the workaround in v1, thanks!

Cheers,

Richard

#2 lore raw
In reply to Xiangyu Chen
Quoted 56 lines · Xiangyu Chen

From: Xiangyu Chen <[email protected]>

Currently, the latest version abseil-cpp contains a new library named "absl_log_internal_format", it's basic package config(.pc file) as below:

prefix=/usr
exec_prefix=${prefix}

......

Requires: absl_config = 20230125, absl_core_headers = 20230125, absl_log_internal_append_truncated = 20230125, absl_log_internal_config = 20230125, absl_log_internal_globals = 20230125, absl_log_severity = 20230125, absl_strings = 20230125, absl_str_format = 20230125, absl_time = 20230125, absl_span = 20230125
......

Normally, the process_pkgconfig() would process variable data before field data in a .pc file, but in the absl_log_internal_format, the field data in "Requires" section contains "xxxx = xxxx" format, the process_pkgconfig() treats them as normal variable and using the setVar() in bitbake's data_smart.py try to process. The absl_log_internal_format field data contains "_append_", this hit the setVar() checking and finally bitbake stop building and reporting an error as below:

"Variable xxx contains an operation using the old override syntax. Please convert this layer/metadata before attempting to use with a newer bitbake."

This patch move the field data process before variable process to avoid the process_pkgconfig() treat the field data as variable.

---

Changes:
V1 -> V2: changing the package.py instead of bitbake's data_smart.py v1 thread please refer:https://lists.openembedded.org/g/bitbake-devel/message/14525

---
 meta/lib/oe/package.py | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)
meta/lib/oe/package.py +0 -6
index c9eb75d852..7a6b31957a 100644
@@ -1823,18 +1823,18 @@ def process_pkgconfig(pkgfiles, d):
                     with open(file, 'r') as f:
                         lines = f.readlines()
                     for l in lines:
-                        m = var_re.match(l)
-                        if m:
-                            name = m.group(1)
-                            val = m.group(2)
-                            pd.setVar(name, pd.expand(val))
-                            continue
                         m = field_re.match(l)
                         if m:
                             hdr = m.group(1)
                             exp = pd.expand(m.group(2))
                             if hdr == 'Requires':

meta/lib/oe/package.py def process_pkgconfig(pkgfiles, d): · line 1840

Quoted 10 lines
meta/lib/oe/package.py +6 -0
                                 pkgconfig_needed[pkg] += exp.replace(',', ' ').split()
+                                continue
+                        m = var_re.match(l)
+                        if m:
+                            name = m.group(1)
+                            val = m.group(2)
+                            pd.setVar(name, pd.expand(val))
 
     for pkg in packages.split():
         pkgs_file = os.path.join(shlibswork_dir, pkg + ".pclist")

It is worth noting this fix actually resolved a number of latent bugs too. This breakage in reproducibility was as a result of this change:

http://autobuilder.yocto.io/pub/repro-fail/oe-reproducible-20230325-0k1j3_4w/packages/diff-html/

which was because the output changed but the sstate hashes didn't. That is because package.py code isn't part of the sstate hashes (currently).

This isn't a fault of this patch. I've a patch in progress to bump various versions to make sure this doesn't break anything further and it does raise a question on whether we should checksum the python functions and track that due to the adverse impact an issue like this can have.

The issue here was missing dependencies, which were missing from -dev packages when there was a version constraint present. I wanted to highlight why fixing things at the root cause is good! If we'd hacked this in bitbake, the other issues would have remained unseen.

Cheers,

Richard