Skip to contentSkip to conversation

bitbake.conf: DEBUG_PREFIX_MAP: move STAGING_DIR_NATIVE before STAGING_DIR_HOST

RFC PATCH Martin Jansa 2 participants 4 messages Sep 4, 2024 2 messages not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

#1 lore raw

Please be aware that:
https://git.openembedded.org/openembedded-core/commit/?id=593ffffc3baf064b982891d61dacebd08aed3e96
claims that these options are evaluated right to left (last one first) and even shows the same issue as what I was seeing as a reason to do the opposite (move STAGING_DIR_NATIVE after STAGING_DIR_HOST) and I'm basically reverting that change (which I didn't noticed until today).

So either the processing order changed since 2018 with gcc-7.3 or something is wrong (at least the comment above DEBUG_PREFIX_MAP would be wrong if the processing order was changed).

Regards,

Quoted 59 lines · Martin Jansa via lists.openembedded.org

* and add -fmacro-prefix-map as well

* the default STAGING_DIR_NATIVE starts with STAGING_DIR_HOST and the only difference is '-native' suffix at the end

* this can lead into replacing STAGING_DIR_NATIVE path with just -native if STAGING_DIR_HOST is listed in DEBUG_PREFIX_MAP before STAGING_DIR_NATIVE

* I've noticed this by accident in python3-matplotlib where buildpaths
QA warning was triggered only for lib32-python3-matplotlib and it was
because pybind11 path to STAGING_DIR_NATIVE was mapped to only
'-native/<path>' in python3-matplotlib build (which doesn't trigger
buildpaths QA and lib32-python3-matplotlib the macro path wasn't
replaced at all, because of 'lib32-' prefix in:
-fdebug-prefix-map=/OE/build/oe-core/tmp-glibc/work/i586-oemllib32-linux/lib32-python3-matplotlib/3.7.2/lib32-recipe-sysroot= \
-fmacro-prefix-map=/OE/build/oe-core/tmp-glibc/work/i586-oemllib32-linux/lib32-python3-matplotlib/3.7.2/lib32-recipe-sysroot= \
-fdebug-prefix-map=/OE/build/oe-core/tmp-glibc/work/i586-oemllib32-linux/lib32-python3-matplotlib/3.7.2/recipe-sysroot-native= \

* more details in meta-python fix for lib32-python3-matplotlib:
https://lists.openembedded.org/g/openembedded-devel/message/112074

---
Sending as RFC, because I haven't build whole world with this and there might be better way to do this or more buildpaths issues triggered by this than fixed

 meta/conf/bitbake.conf | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
meta/conf/bitbake.conf +2 -2
index d8252c5b82..ad152ffb6a 100644
@@ -653,9 +653,10 @@ DEBUG_PREFIX_MAP ?= "-fcanon-prefix-map \
  -fdebug-prefix-map=${S}=${TARGET_DBGSRC_DIR} \
  -fmacro-prefix-map=${B}=${TARGET_DBGSRC_DIR} \
  -fdebug-prefix-map=${B}=${TARGET_DBGSRC_DIR} \
+ -fdebug-prefix-map=${STAGING_DIR_NATIVE}= \
+ -fmacro-prefix-map=${STAGING_DIR_NATIVE}= \
  -fdebug-prefix-map=${STAGING_DIR_HOST}= \
  -fmacro-prefix-map=${STAGING_DIR_HOST}= \
- -fdebug-prefix-map=${STAGING_DIR_NATIVE}= \
 "
 DEBUG_LEVELFLAG ?= "-g"
 
--

2.46.0

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#204110): https://lists.openembedded.org/g/openembedded-core/message/204110
Mute This Topic: https://lists.openembedded.org/mt/108229773/3617156
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

A message that is not in the archive (sent off-list or not yet received): [email protected]

#3 lore raw

Some related info I've found now:
https://reviews.llvm.org/D148975?id=516863
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109591

maybe affected by -fcanon-prefix-map used since gcc-13:
https://git.openembedded.org/openembedded-core/commit/?id=abe97fa74b10c2c71d79bc3573d4edb509b111f5
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=108464#c8

and I was also using ccache which used to have some issues in this area (but the python3-matplotlib issue is IIRC reproducible without ccache as well):
https://github.com/ccache/ccache/issues/163

We might need more experiments before this is applied (I'm glad I've sent it only as RFC). I'll try to find some time next week.

Quoted 78 lines · Martin Jansa via lists.openembedded.org

Please be aware that:
https://git.openembedded.org/openembedded-core/commit/?id=593ffffc3baf064b982891d61dacebd08aed3e96
claims that these options are evaluated right to left (last one first) and even shows the same issue as what I was seeing as a reason to do the opposite (move STAGING_DIR_NATIVE after STAGING_DIR_HOST) and I'm basically reverting that change (which I didn't noticed until today).

So either the processing order changed since 2018 with gcc-7.3 or something is wrong (at least the comment above DEBUG_PREFIX_MAP would be wrong if the processing order was changed).

Regards,

Quoted 48 lines · Martin Jansa via lists.openembedded.org

* and add -fmacro-prefix-map as well

* the default STAGING_DIR_NATIVE starts with STAGING_DIR_HOST and the only difference is '-native' suffix at the end

* this can lead into replacing STAGING_DIR_NATIVE path with just -native if STAGING_DIR_HOST is listed in DEBUG_PREFIX_MAP before STAGING_DIR_NATIVE

* I've noticed this by accident in python3-matplotlib where buildpaths
QA warning was triggered only for lib32-python3-matplotlib and it was
because pybind11 path to STAGING_DIR_NATIVE was mapped to only
'-native/<path>' in python3-matplotlib build (which doesn't trigger
buildpaths QA and lib32-python3-matplotlib the macro path wasn't
replaced at all, because of 'lib32-' prefix in:
-fdebug-prefix-map=/OE/build/oe-core/tmp-glibc/work/i586-oemllib32-linux/lib32-python3-matplotlib/3.7.2/lib32-recipe-sysroot= \
-fmacro-prefix-map=/OE/build/oe-core/tmp-glibc/work/i586-oemllib32-linux/lib32-python3-matplotlib/3.7.2/lib32-recipe-sysroot= \
-fdebug-prefix-map=/OE/build/oe-core/tmp-glibc/work/i586-oemllib32-linux/lib32-python3-matplotlib/3.7.2/recipe-sysroot-native= \

* more details in meta-python fix for lib32-python3-matplotlib:
https://lists.openembedded.org/g/openembedded-devel/message/112074

---
Sending as RFC, because I haven't build whole world with this and there might be better way to do this or more buildpaths issues triggered by this than fixed

 meta/conf/bitbake.conf | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
meta/conf/bitbake.conf +2 -2
index d8252c5b82..ad152ffb6a 100644
@@ -653,9 +653,10 @@ DEBUG_PREFIX_MAP ?= "-fcanon-prefix-map \
  -fdebug-prefix-map=${S}=${TARGET_DBGSRC_DIR} \
  -fmacro-prefix-map=${B}=${TARGET_DBGSRC_DIR} \
  -fdebug-prefix-map=${B}=${TARGET_DBGSRC_DIR} \
+ -fdebug-prefix-map=${STAGING_DIR_NATIVE}= \
+ -fmacro-prefix-map=${STAGING_DIR_NATIVE}= \
  -fdebug-prefix-map=${STAGING_DIR_HOST}= \
  -fmacro-prefix-map=${STAGING_DIR_HOST}= \
- -fdebug-prefix-map=${STAGING_DIR_NATIVE}= \
 "
 DEBUG_LEVELFLAG ?= "-g"
 
--
Quoted 5 lines

2.46.0

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#204190): https://lists.openembedded.org/g/openembedded-core/message/204190
Mute This Topic: https://lists.openembedded.org/mt/108229773/3617156
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#4 lore raw
Quoted 8 lines · Martin Jansa via lists.openembedded.org

Some related info I've found now:
https://reviews.llvm.org/D148975?id=516863
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109591

maybe affected by -fcanon-prefix-map used since gcc-13:
https://git.openembedded.org/openembedded-core/commit/?id=abe97fa74b10c2c71d79bc3573d4edb509b111f5
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=108464#c8

Quoted 7 lines

and I was also using ccache which used to have some issues in this area (but the python3-matplotlib issue is IIRC reproducible without ccache as well):
https://github.com/ccache/ccache/issues/163

We might need more experiments before this is applied (I'm glad I've sent it only as RFC). I'll try to find some time next week.

I'm a bit worried that gcc might apply the ordering one way in some locations and the other way around in others so there may be a bug lurking somewhere. I agree we should look further into this.

Cheers,

Richard

#5 lore raw
Quoted 16 lines · Richard Purdie
Quoted 13 lines · Martin Jansa via lists.openembedded.org

Some related info I've found now:
https://reviews.llvm.org/D148975?id=516863
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109591

maybe affected by -fcanon-prefix-map used since gcc-13:
https://git.openembedded.org/openembedded-core/commit/?id=abe97fa74b10c2c71d79bc3573d4edb509b111f5
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=108464#c8

and I was also using ccache which used to have some issues in this area (but the python3-matplotlib issue is IIRC reproducible without ccache as well):
https://github.com/ccache/ccache/issues/163

Quoted 6 lines
Quoted 2 lines

We might need more experiments before this is applied (I'm glad I've sent it only as RFC). I'll try to find some time next week.

I'm a bit worried that gcc might apply the ordering one way in some locations and the other way around in others so there may be a bug lurking somewhere. I agree we should look further into this.

I had only a bit of time in the morning to quickly scan through them, but if it's correct that gcc as well as llvm are applying the longest substitution now, then the ordering might not be important anymore and it might actually explain the case with python3-matplotlib I was seeing, because for -fmacro-prefix-map there wasn't the native path at all, so it "correctly" replaced the longest matching one which was -fmacro-prefix-map=${STAGING_DIR_HOST}= for python3-matplotlib and no replacement for lib32-python3-matplotlib as it didn't match at all in any of them.

Will redo both builds with just -fmacro-prefix-map=${STAGING_DIR_NATIVE}= added at the end to see if it fixes paths in both python3-matplotlib and lib32-python3-matplotlib and should probably test the same with clang to be sure it works the same.

Regards,