Skip to contentSkip to conversation

glib-2.0: call os.path.normpath on THISDIR

PATCH In review Martin Jansa 3 participants 8 messages Mar 2, 2021 – Mar 5, 2021 1 message not in the archive

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 glib-2.0: call os.path.normpath on THISDIR

* some build environments have relative paths in THISDIR, e.g. from OEROOT set in:
https://github.com/96boards/oe-rpb-manifest/blob/1e3345c26c56f77f3a15a3978f412a25955d2606/conf/bblayers.conf#L4
and then the paths in filename normalized in:
filename = os.path.normpath(os.path.join(path, meson.cross.d, element)) don't match.

* COREBASE used here before didn't have this issue because the value is already normalized when set in:
meta/conf/layer.conf:COREBASE = '${@os.path.normpath("${LAYERDIR}/../")}'

---
 meta/recipes-core/glib-2.0/glib-2.0_2.66.7.bb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/recipes-core/glib-2.0/glib-2.0_2.66.7.bb +1 -1
index 882a89da7a..a2fa5345b4 100644
@@ -31,7 +31,7 @@ def find_meson_cross_files(d):
     if bb.data.inherits_class('native', d):
         return ""
 
-    thisdir = d.getVar("THISDIR")
+    thisdir = os.path.normpath(d.getVar("THISDIR"))
     import collections
     sitedata = siteinfo_data(d)
     # filename -> found
Signature · 1 line
2.30.0
#1 lore raw
Quoted 15 lines

-----Original Message-----
From: [email protected] <openembedded- [email protected]> On Behalf Of Martin Jansa
Sent: den 2 mars 2021 20:32
To: [email protected]
Cc: [email protected]; [email protected]; Martin Jansa <[email protected]>
Subject: [OE-core] [PATCH] glib-2.0: call os.path.normpath on THISDIR

* some build environments have relative paths in THISDIR, e.g. from OEROOT set in:
https://github.com/96boards/oe-rpb-manifest/blob/1e3345c26c56f77f3a15a3978f412a25955d2606/conf/bblayers.conf#L4
and then the paths in filename normalized in:
filename = os.path.normpath(os.path.join(path, meson.cross.d, element)) don't match.

Quoted 3 lines

* COREBASE used here before didn't have this issue because the value is already normalized when set in:
meta/conf/layer.conf:COREBASE = '${@os.path.normpath("${LAYERDIR}/../")}'

That is not the only problem with the change to use ${THISDIR}. We get a million errors because we have bbappend files for glib-2.0. This is because bitbake will now give an error for any potential cross-files from those layers, regardless if they exist or not (they don't).

I will send a patch that I believe solves this in a way that also avoids the need for the bb.error().

//Peter

Quoted 22 lines
---
 meta/recipes-core/glib-2.0/glib-2.0_2.66.7.bb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/recipes- renamed from meta/recipes-core/glib-2.0/glib-2.0_2.66.7.bb +0 -0

core/glib-2.0/glib-2.0_2.66.7.bb
index 882a89da7a..a2fa5345b4 100644

meta/recipes-core/glib-2.0/glib-2.0_2.66.7.bb +1 -2
@@ -31,7 +31,7 @@ def find_meson_cross_files(d):
     if bb.data.inherits_class('native', d):
         return ""
 
-    thisdir = d.getVar("THISDIR")
+    thisdir = os.path.normpath(d.getVar("THISDIR"))
     import collections
     sitedata = siteinfo_data(d)
     # filename -> found
--

2.30.0

A message that is not in the archive (sent off-list or not yet received): [email protected]

#3 lore raw
Quoted 30 lines

-----Original Message-----
From: [email protected] <[email protected]> On Behalf Of Peter Kjellerstedt
Sent: den 5 mars 2021 11:54
To: Martin Jansa <[email protected]>; [email protected]
Cc: [email protected]; [email protected]
Subject: Re: [OE-core] [PATCH] glib-2.0: call os.path.normpath on THISDIR

Quoted 13 lines

-----Original Message-----
From: [email protected] <[email protected]> On Behalf Of Martin Jansa
Sent: den 2 mars 2021 20:32
To: [email protected]
Cc: [email protected]; [email protected]; Martin Jansa <[email protected]>
Subject: [OE-core] [PATCH] glib-2.0: call os.path.normpath on THISDIR

* some build environments have relative paths in THISDIR, e.g. from OEROOT set in:
https://github.com/96boards/oe-rpb-manifest/blob/1e3345c26c56f77f3a15a3978f412a25955d2606/conf/bblayers.conf#L4
and then the paths in filename normalized in:
filename = os.path.normpath(os.path.join(path, meson.cross.d, element)) don't match.

Quoted 3 lines

* COREBASE used here before didn't have this issue because the value is already normalized when set in:
meta/conf/layer.conf:COREBASE =

'${@os.path.normpath("${LAYERDIR}/../")}'

That is not the only problem with the change to use ${THISDIR}. We get a million errors because we have bbappend files for glib-2.0. This is because bitbake will now give an error for any potential cross-files from those layers, regardless if they exist or not (they don't).

Quoted 4 lines

I will send a patch that I believe solves this in a way that also avoids the need for the bb.error().

//Peter

Actually, thinking a bit more on this, I believe that with my patch applied, this patch should _not_ be applied. That is because it makes the paths relative to thisdir, but then uses ${THISDIR} when using them. Thus if thisdir is normalized, it may no longer match ${THISDIR} and the relative paths will not work as intended.

//Peter

Quoted 22 lines

Review

Quoted 6 lines
---
 meta/recipes-core/glib-2.0/glib-2.0_2.66.7.bb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
(quoted diff) +0 -0

b/meta/recipes-

Quoted 15 lines

core/glib-2.0/glib-2.0_2.66.7.bb
index 882a89da7a..a2fa5345b4 100644

meta/recipes-core/glib-2.0/glib-2.0_2.66.7.bb +1 -2
@@ -31,7 +31,7 @@ def find_meson_cross_files(d):
     if bb.data.inherits_class('native', d):
         return ""
 
-    thisdir = d.getVar("THISDIR")
+    thisdir = os.path.normpath(d.getVar("THISDIR"))
     import collections
     sitedata = siteinfo_data(d)
     # filename -> found
--

2.30.0

#4 lore raw
Quoted 33 lines · Peter Kjellerstedt
Quoted 33 lines

-----Original Message-----
From: [email protected] <[email protected]> On Behalf Of Peter Kjellerstedt
Sent: den 5 mars 2021 11:54
To: Martin Jansa <[email protected]>; [email protected]
Cc: [email protected]; [email protected]
Subject: Re: [OE-core] [PATCH] glib-2.0: call os.path.normpath on THISDIR

Quoted 13 lines

-----Original Message-----
From: [email protected] <[email protected]> On Behalf Of Martin Jansa
Sent: den 2 mars 2021 20:32
To: [email protected]
Cc: [email protected]; [email protected]; Martin Jansa <[email protected]>
Subject: [OE-core] [PATCH] glib-2.0: call os.path.normpath on THISDIR

* some build environments have relative paths in THISDIR, e.g. from OEROOT set in:
https://github.com/96boards/oe-rpb-manifest/blob/1e3345c26c56f77f3a15a3978f412a25955d2606/conf/bblayers.conf#L4
and then the paths in filename normalized in:
filename = os.path.normpath(os.path.join(path, meson.cross.d, element)) don't match.

Quoted 3 lines

* COREBASE used here before didn't have this issue because the value is already normalized when set in:
meta/conf/layer.conf:COREBASE =

'${@os.path.normpath("${LAYERDIR}/../")}'

That is not the only problem with the change to use ${THISDIR}. We get a million errors because we have bbappend files for glib-2.0. This is because bitbake will now give an error for any potential cross-files from those layers, regardless if they exist or not (they don't).

I will send a patch that I believe solves this in a way that also avoids the need for the bb.error().

Quoted 7 lines
Quoted 1 line

//Peter

Actually, thinking a bit more on this, I believe that with my patch applied, this patch should _not_ be applied. That is because it makes the paths relative to thisdir, but then uses ${THISDIR} when using them. Thus if thisdir is normalized, it may no longer match ${THISDIR} and the relative paths will not work as intended.

I'm sorry I've tested and wrote that part of commit message when I had if os.path.exists(filename):
at the beginning of this whole section, which I've then re-wrote to just store only existing filenames in files as a list instead of a map (to simplify the rest of this function as well), but then I was wondering why Ross used the map in the initial implementation here:
https://git.openembedded.org/openembedded-core/commit/?id=5acd9cbc9d5c6355010775250fb25f043441c5cd
and assumed it was needed to get all possible filenames in do_configure[file-checksums] in case they are created later.

So I've removed all my cleanups and returned to smallest change to just replace COREBASE with THISDIR.

What about checking the existence before showing the error, that will show an error only when matching file exists while keeping all not-skipped paths in do_configure[file-checksums] even when they don't exist yet.

Will send a patch for this shortly.

Quoted 26 lines

//Peter

Quoted 22 lines

Review

Quoted 6 lines
---
 meta/recipes-core/glib-2.0/glib-2.0_2.66.7.bb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
(quoted diff) +0 -0

b/meta/recipes-

Quoted 15 lines

core/glib-2.0/glib-2.0_2.66.7.bb
index 882a89da7a..a2fa5345b4 100644

meta/recipes-core/glib-2.0/glib-2.0_2.66.7.bb +1 -2
@@ -31,7 +31,7 @@ def find_meson_cross_files(d):
     if bb.data.inherits_class('native', d):
         return ""
 
-    thisdir = d.getVar("THISDIR")
+    thisdir = os.path.normpath(d.getVar("THISDIR"))
     import collections
     sitedata = siteinfo_data(d)
     # filename -> found
--

2.30.0

#5 lore raw
Quoted 75 lines

-----Original Message-----
From: Martin Jansa <[email protected]>
Sent: den 5 mars 2021 12:51
To: Peter Kjellerstedt <[email protected]>
Cc: [email protected]; [email protected]; [email protected]
Subject: Re: [OE-core] [PATCH] glib-2.0: call os.path.normpath on THISDIR

Quoted 2 lines · Peter Kjellerstedt
Quoted 2 lines

-----Original Message-----
From: [email protected] <openembedded-

[email protected]> On Behalf Of Peter Kjellerstedt

Quoted 2 lines
Quoted 2 lines

Sent: den 5 mars 2021 11:54
To: Martin Jansa <[email protected]>; openembedded-

[email protected]

Quoted 2 lines
Quoted 2 lines

Cc: [email protected]; [email protected]
Subject: Re: [OE-core] [PATCH] glib-2.0: call os.path.normpath on

THISDIR

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

-----Original Message-----
From: [email protected] <openembedded-

[email protected]> On Behalf Of Martin Jansa

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Sent: den 2 mars 2021 20:32
To: [email protected]
Cc: [email protected]; [email protected]; Martin Jansa

<[email protected]>

Quoted 1 line
Quoted 1 line
Quoted 1 line

Subject: [OE-core] [PATCH] glib-2.0: call os.path.normpath on

THISDIR

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

* some build environments have relative paths in THISDIR, e.g. from

OEROOT set in:

Quoted 1 line
Quoted 1 line
Quoted 1 line

https://github.com/96boards/oe-rpb-

manifest/blob/1e3345c26c56f77f3a15a3978f412a25955d2606/conf/bblayers.conf# L4

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

and then the paths in filename normalized in:
filename = os.path.normpath(os.path.join(path, meson.cross.d,

element))

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

don't match.

* COREBASE used here before didn't have this issue because the value

is already

Quoted 12 lines
Quoted 12 lines
Quoted 2 lines

normalized when set in:
meta/conf/layer.conf:COREBASE =

'${@os.path.normpath("${LAYERDIR}/../")}'

That is not the only problem with the change to use ${THISDIR}. We get a million errors because we have bbappend files for glib-2.0. This is because bitbake will now give an error for any potential cross-files from those layers, regardless if they exist or not (they don't).

I will send a patch that I believe solves this in a way that also avoids the need for the bb.error().

Quoted 7 lines
Quoted 1 line

//Peter

Actually, thinking a bit more on this, I believe that with my patch applied, this patch should _not_ be applied. That is because it makes the paths relative to thisdir, but then uses ${THISDIR} when using them. Thus if thisdir is normalized, it may no longer match ${THISDIR} and the relative paths will not work as intended.

I'm sorry I've tested and wrote that part of commit message when I had if os.path.exists(filename):
at the beginning of this whole section, which I've then re-wrote to just store only existing filenames in files as a list instead of a map (to simplify the rest of this function as well), but then I was wondering why Ross used the map in the initial implementation here:
https://git.openembedded.org/openembedded-
core/commit/?id=5acd9cbc9d5c6355010775250fb25f043441c5cd
and assumed it was needed to get all possible filenames in do_configure[file-checksums] in case they are created later.

So I've removed all my cleanups and returned to smallest change to just replace COREBASE with THISDIR.

Quoted 6 lines

What about checking the existence before showing the error, that will show an error only when matching file exists while keeping all not-skipped paths in do_configure[file-checksums] even when they don't exist yet.

Will send a patch for this shortly.

Any objections to my patch, where there is no need for the bb.error() in the first place?

//Peter

#6 lore raw
Quoted 31 lines · Peter Kjellerstedt
Quoted 31 lines

-----Original Message-----
From: Martin Jansa <[email protected]>
Sent: den 5 mars 2021 12:51
To: Peter Kjellerstedt <[email protected]>
Cc: [email protected]; [email protected];

Quoted 6 lines

Actually, thinking a bit more on this, I believe that with my patch applied, this patch should _not_ be applied. That is because it makes the paths relative to thisdir, but then uses ${THISDIR} when using them. Thus if thisdir is normalized, it may no longer match ${THISDIR} and the relative paths will not work as intended.

I'm sorry I've tested and wrote that part of commit message when I had if os.path.exists(filename):
at the beginning of this whole section, which I've then re-wrote to just store only existing filenames in files as a list instead of a map (to simplify the rest of this function as well), but then I was wondering why Ross used the map in the initial implementation here:
https://git.openembedded.org/openembedded-
core/commit/?id=5acd9cbc9d5c6355010775250fb25f043441c5cd
and assumed it was needed to get all possible filenames in do_configure[file-checksums] in case they are created later.

So I've removed all my cleanups and returned to smallest change to just replace COREBASE with THISDIR.

What about checking the existence before showing the error, that will show an error only when matching file exists while keeping all not-skipped paths in do_configure[file-checksums] even when they don't exist yet.

Quoted 4 lines
Quoted 1 line

Will send a patch for this shortly.

Any objections to my patch, where there is no need for the bb.error() in the first place?

Your patch encodes the paths of layers relative to each other into the sstate checksums. I'm not sure we want to encourage that and this code will end up copy and pasted around without people realising what they're doing :/.

Cheers,

Richard

#7 lore raw
Quoted 47 lines

-----Original Message-----
From: Richard Purdie <[email protected]>
Sent: den 5 mars 2021 18:54
To: Peter Kjellerstedt <[email protected]>; Martin Jansa <[email protected]>
Cc: [email protected]; [email protected]; [email protected]
Subject: Re: [OE-core] [PATCH] glib-2.0: call os.path.normpath on THISDIR

Quoted 8 lines · Peter Kjellerstedt
Quoted 8 lines

-----Original Message-----
From: Martin Jansa <[email protected]>
Sent: den 5 mars 2021 12:51
To: Peter Kjellerstedt <[email protected]>
Cc: [email protected]; [email protected];

Quoted 3 lines

Actually, thinking a bit more on this, I believe that with my patch applied, this patch should _not_ be applied. That is because it

makes

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

the paths relative to thisdir, but then uses ${THISDIR} when using them. Thus if thisdir is normalized, it may no longer match

${THISDIR}

Quoted 5 lines
Quoted 5 lines
Quoted 1 line

and the relative paths will not work as intended.

I'm sorry I've tested and wrote that part of commit message when I had if os.path.exists(filename):
at the beginning of this whole section, which I've then re-wrote to

just

Quoted 9 lines
Quoted 9 lines

store only existing filenames in files as a list instead of a map (to simplify the rest of this function as well), but then I was wondering why Ross used the map in the initial implementation here:
https://git.openembedded.org/openembedded-
core/commit/?id=5acd9cbc9d5c6355010775250fb25f043441c5cd and assumed it was needed to get all possible filenames in do_configure[file-checksums] in case they are created later.

So I've removed all my cleanups and returned to smallest change to

just

Quoted 9 lines
Quoted 8 lines

replace COREBASE with THISDIR.

What about checking the existence before showing the error, that will show an error only when matching file exists while keeping all not-skipped paths in do_configure[file-checksums] even when they don't exist yet.

Will send a patch for this shortly.

Quoted 7 lines
Quoted 2 lines

Any objections to my patch, where there is no need for the bb.error() in the first place?

Your patch encodes the paths of layers relative to each other into the sstate checksums. I'm not sure we want to encourage that and this code will end up copy and pasted around without people realising what they're doing :/.

Hmm, I guess that may be considered bad. :/

Quoted 3 lines

Cheers,

Richard

Then we will have to decide on what to do with the cross-files. Currently the code tries to handle cross-files provided by any layer, but with the recent changes and without mine it really only supports specifying cross-files in the same layer where the recipe is. So we will either have to find a way to actually support cross-files from multiple layers, or we may as well rip out the current attempt at supporting it.

//Peter

#8 lore raw
Quoted 55 lines · Peter Kjellerstedt
Quoted 47 lines

-----Original Message-----
From: Richard Purdie <[email protected]>
Sent: den 5 mars 2021 18:54
To: Peter Kjellerstedt <[email protected]>; Martin Jansa <[email protected]>
Cc: [email protected]; [email protected]; [email protected]
Subject: Re: [OE-core] [PATCH] glib-2.0: call os.path.normpath on THISDIR

Quoted 8 lines · Peter Kjellerstedt
Quoted 8 lines

-----Original Message-----
From: Martin Jansa <[email protected]>
Sent: den 5 mars 2021 12:51
To: Peter Kjellerstedt <[email protected]>
Cc: [email protected]; [email protected];

Quoted 3 lines

Actually, thinking a bit more on this, I believe that with my patch applied, this patch should _not_ be applied. That is because it

makes

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

the paths relative to thisdir, but then uses ${THISDIR} when using them. Thus if thisdir is normalized, it may no longer match

${THISDIR}

Quoted 5 lines
Quoted 5 lines
Quoted 1 line

and the relative paths will not work as intended.

I'm sorry I've tested and wrote that part of commit message when I had if os.path.exists(filename):
at the beginning of this whole section, which I've then re-wrote to

just

Quoted 9 lines
Quoted 9 lines

store only existing filenames in files as a list instead of a map (to simplify the rest of this function as well), but then I was wondering why Ross used the map in the initial implementation here:
https://git.openembedded.org/openembedded-
core/commit/?id=5acd9cbc9d5c6355010775250fb25f043441c5cd and assumed it was needed to get all possible filenames in do_configure[file-checksums] in case they are created later.

So I've removed all my cleanups and returned to smallest change to

just

Quoted 9 lines
Quoted 8 lines

replace COREBASE with THISDIR.

What about checking the existence before showing the error, that will show an error only when matching file exists while keeping all not-skipped paths in do_configure[file-checksums] even when they don't exist yet.

Will send a patch for this shortly.

Quoted 7 lines
Quoted 2 lines

Any objections to my patch, where there is no need for the bb.error() in the first place?

Your patch encodes the paths of layers relative to each other into the sstate checksums. I'm not sure we want to encourage that and this code will end up copy and pasted around without people realising what they're doing :/.

Quoted 8 lines

Hmm, I guess that may be considered bad. :/

Quoted 3 lines

Cheers,

Richard

Then we will have to decide on what to do with the cross-files. Currently the code tries to handle cross-files provided by any layer, but with the

I wouldn't say that by using FILESPATH it really tried to handle cross-files from other layers.

Quoted 4 lines

recent changes and without mine it really only supports specifying cross-files in the same layer where the recipe is. So we will either have to find a way to actually support cross-files from multiple layers, or we may as well rip out the current attempt at supporting it.

And replace it with what?
https://git.openembedded.org/openembedded-core/commit/?id=5acd9cbc9d5c6355010775250fb25f043441c5cd
wasn't added to support cross-files from other layers, so we still need this for the cross-files in oe-core, right?

Now it will just refuse to consider cross-files when they are outside oe-core while showing an error, instead of silently making signatures of glib-2.0 and everything depending specific to the build path where it was built.

And as small benefit of using THISDIR it will also work when you copy glib-2.0 and all its cross-files to some other layer (like somebody did @LGE when backporting newer glib-2.0).