Skip to contentSkip to conversation

.templateconf: Mark "meta/conf" as an invalid path for templates

PATCH In review Peter Kjellerstedt 4 participants 26 messages Sep 22, 2022 – Sep 28, 2022 3 messages not in the archive

lore
Share & tools
lore
#0 lore raw

PATCH 1/2 .templateconf: Mark "meta/conf" as an invalid path for templates

Previously this was hardcoded in oe-setup-builddir.

---
 .templateconf | 1 +
 1 file changed, 1 insertion(+)
.templateconf +1 -0
index e399e06acd..8e4a9ebdc6 100644
@@ -1,2 +1,3 @@
 # Template settings
 TEMPLATECONF=${TEMPLATECONF:-meta/conf/templates/default}
+INVALID_TEMPLATECONFS=meta/conf
#1 lore raw

PATCH 2/2 oe-setup-builddir: Do not hardcode invalid paths for templates

Previously, the paths "meta/conf" and "meta-poky/conf" were hardcoded as invalid paths for templates. However, this is suboptimal for other distros that are setup similarly to Poky. Instead, add support for a new variable INVALID_TEMPLATECONFS, which takes a list of invalid paths. It is expected that this variable is set in the .templateconf file together with the default value for TEMPLATECONF, typically at the same time that TEMPLATECONF is updated to match the new requirements.

---
 scripts/oe-setup-builddir | 21 +++++++++++++--------
 1 file changed, 13 insertions(+), 8 deletions(-)
scripts/oe-setup-builddir +13 -8
index 225919be92..c1148daf4f 100755
@@ -34,14 +34,19 @@ chmod -st "$BUILDDIR/conf" 2>/dev/null || echo "WARNING: unable to chmod $BUILDD
 
 cd "$BUILDDIR" || die "Failed to change directory to $BUILDDIR!"
 
-if [ -z "$TEMPLATECONF" ] && [ -f "$BUILDDIR/conf/templateconf.cfg" ]; then
-    TEMPLATECONF=$(cat "$BUILDDIR/conf/templateconf.cfg")
-    # The following two are no longer valid; unsetting them will automatically get them replaced
-    # with correct ones.
-    if [ "$TEMPLATECONF" = meta/conf ] || [ "$TEMPLATECONF" = meta-poky/conf ]; then
-        unset TEMPLATECONF
-        rm "$BUILDDIR/conf/templateconf.cfg"
-    fi
+TEMPLATECONF_CFG="$BUILDDIR/conf/templateconf.cfg"
+if [ -z "$TEMPLATECONF" ] && [ -f "$TEMPLATECONF_CFG" ]; then
+    TEMPLATECONF=$(cat "$TEMPLATECONF_CFG")
+    # Unset TEMPLATECONF if it is set to a known invalid value to have it
+    # automatically replaced with a correct one.
+    for dir in $INVALID_TEMPLATECONFS; do
+        if [ "$TEMPLATECONF" = "$dir" ]; then
+           echo "WARNING: Removing $TEMPLATECONF_CFG as it contained the invalid value '$TEMPLATECONF'"
+           unset TEMPLATECONF
+           rm "$TEMPLATECONF_CFG"
+	   break;
+        fi
+    done
 fi
 
 . "$OEROOT/.templateconf"
#2 lore raw

Re: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode invalid paths for templates

Quoted 34 lines · Peter Kjellerstedt

Previously, the paths "meta/conf" and "meta-poky/conf" were hardcoded as invalid paths for templates. However, this is suboptimal for other distros that are setup similarly to Poky. Instead, add support for a new variable INVALID_TEMPLATECONFS, which takes a list of invalid paths. It is expected that this variable is set in the .templateconf file together with the default value for TEMPLATECONF, typically at the same time that TEMPLATECONF is updated to match the new requirements.

---
 scripts/oe-setup-builddir | 21 +++++++++++++--------
 1 file changed, 13 insertions(+), 8 deletions(-)
scripts/oe-setup-builddir +4 -8
index 225919be92..c1148daf4f 100755
@@ -34,14 +34,19 @@ chmod -st "$BUILDDIR/conf" 2>/dev/null || echo "WARNING: unable to chmod $BUILDD
 
 cd "$BUILDDIR" || die "Failed to change directory to $BUILDDIR!"
 
-if [ -z "$TEMPLATECONF" ] && [ -f "$BUILDDIR/conf/templateconf.cfg" ]; then
-    TEMPLATECONF=$(cat "$BUILDDIR/conf/templateconf.cfg")
-    # The following two are no longer valid; unsetting them will automatically get them replaced
-    # with correct ones.
-    if [ "$TEMPLATECONF" = meta/conf ] || [ "$TEMPLATECONF" = meta-poky/conf ]; then
-        unset TEMPLATECONF
-        rm "$BUILDDIR/conf/templateconf.cfg"
-    fi
+TEMPLATECONF_CFG="$BUILDDIR/conf/templateconf.cfg"
+if [ -z "$TEMPLATECONF" ] && [ -f "$TEMPLATECONF_CFG" ]; then
+    TEMPLATECONF=$(cat "$TEMPLATECONF_CFG")
+    # Unset TEMPLATECONF if it is set to a known invalid value to have it

scripts/oe-setup-builddir chmod -st "$BUILDDIR/conf" 2>/dev/null || echo "WARNING: unable to chmod $BUILDD · line 50

Quoted 10 lines
scripts/oe-setup-builddir +9 -0
+    # automatically replaced with a correct one.
+    for dir in $INVALID_TEMPLATECONFS; do
+        if [ "$TEMPLATECONF" = "$dir" ]; then
+           echo "WARNING: Removing $TEMPLATECONF_CFG as it contained the invalid value '$TEMPLATECONF'"
+           unset TEMPLATECONF
+           rm "$TEMPLATECONF_CFG"
+	   break;
+        fi
+    done
 fi

I've been trying to keep out this discussion but I really don't like piling in more complexity and more things we have to support because of a legacy code path we're trying to obsolete.

I also suspect this is going to be used to "subvert" the move to try and standardise and if that happens, we may as well not have bothered making changes at all :(.

As such I'm worried about this direction.

Cheers,

Richard

#3 lore raw

RE: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode invalid paths for templates

Quoted 60 lines

-----Original Message-----
From: Richard Purdie <[email protected]>
Sent: den 22 september 2022 15:14
To: Peter Kjellerstedt <[email protected]>; [email protected]
Subject: Re: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode invalid paths for templates

Quoted 34 lines · Peter Kjellerstedt

Previously, the paths "meta/conf" and "meta-poky/conf" were hardcoded as invalid paths for templates. However, this is suboptimal for other distros that are setup similarly to Poky. Instead, add support for a new variable INVALID_TEMPLATECONFS, which takes a list of invalid paths. It is expected that this variable is set in the .templateconf file together with the default value for TEMPLATECONF, typically at the same time that TEMPLATECONF is updated to match the new requirements.

---
 scripts/oe-setup-builddir | 21 +++++++++++++--------
 1 file changed, 13 insertions(+), 8 deletions(-)
scripts/oe-setup-builddir +4 -8
index 225919be92..c1148daf4f 100755
@@ -34,14 +34,19 @@ chmod -st "$BUILDDIR/conf" 2>/dev/null || echo "WARNING: unable to chmod $BUILDD
 
 cd "$BUILDDIR" || die "Failed to change directory to $BUILDDIR!"
 
-if [ -z "$TEMPLATECONF" ] && [ -f "$BUILDDIR/conf/templateconf.cfg" ]; then
-    TEMPLATECONF=$(cat "$BUILDDIR/conf/templateconf.cfg")
-    # The following two are no longer valid; unsetting them will automatically get them replaced
-    # with correct ones.
-    if [ "$TEMPLATECONF" = meta/conf ] || [ "$TEMPLATECONF" = meta-poky/conf ]; then
-        unset TEMPLATECONF
-        rm "$BUILDDIR/conf/templateconf.cfg"
-    fi
+TEMPLATECONF_CFG="$BUILDDIR/conf/templateconf.cfg"
+if [ -z "$TEMPLATECONF" ] && [ -f "$TEMPLATECONF_CFG" ]; then
+    TEMPLATECONF=$(cat "$TEMPLATECONF_CFG")
+    # Unset TEMPLATECONF if it is set to a known invalid value to have it

scripts/oe-setup-builddir chmod -st "$BUILDDIR/conf" 2>/dev/null || echo "WARNING: unable to chmod $BUILDD · line 50

Quoted 10 lines
scripts/oe-setup-builddir +9 -0
+    # automatically replaced with a correct one.
+    for dir in $INVALID_TEMPLATECONFS; do
+        if [ "$TEMPLATECONF" = "$dir" ]; then
+           echo "WARNING: Removing $TEMPLATECONF_CFG as it contained the invalid value '$TEMPLATECONF'"
+           unset TEMPLATECONF
+           rm "$TEMPLATECONF_CFG"
+	   break;
+        fi
+    done
 fi

I've been trying to keep out this discussion but I really don't like piling in more complexity and more things we have to support because of a legacy code path we're trying to obsolete.

I also suspect this is going to be used to "subvert" the move to try and standardise and if that happens, we may as well not have bothered making changes at all :(.

Quoted 5 lines

As such I'm worried about this direction.

Cheers,

Richard

All I want is to remove the hardcoded superficial limitations and allow us that provide distros setup just as Poky is setup to continue to do so.

//Peter

#4 lore raw

Re: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode invalid paths for templates

.templateconf is not the way to specify the location of custom templates, and never was. It got added to documentation by misunderstanding, this has now been removed, and so you need to update your tooling so that templates are set with TEMPLATECONF going forward.

Alex

Quoted 82 lines · Peter Kjellerstedt
Quoted 61 lines

-----Original Message-----
From: Richard Purdie <[email protected]>
Sent: den 22 september 2022 15:14
To: Peter Kjellerstedt <[email protected]>; [email protected]
Subject: Re: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode invalid paths for templates

Quoted 34 lines · Peter Kjellerstedt

Previously, the paths "meta/conf" and "meta-poky/conf" were hardcoded as invalid paths for templates. However, this is suboptimal for other distros that are setup similarly to Poky. Instead, add support for a new variable INVALID_TEMPLATECONFS, which takes a list of invalid paths. It is expected that this variable is set in the .templateconf file together with the default value for TEMPLATECONF, typically at the same time that TEMPLATECONF is updated to match the new requirements.

---
 scripts/oe-setup-builddir | 21 +++++++++++++--------
 1 file changed, 13 insertions(+), 8 deletions(-)
scripts/oe-setup-builddir +4 -8
index 225919be92..c1148daf4f 100755
@@ -34,14 +34,19 @@ chmod -st "$BUILDDIR/conf" 2>/dev/null || echo "WARNING: unable to chmod $BUILDD
 
 cd "$BUILDDIR" || die "Failed to change directory to $BUILDDIR!"
 
-if [ -z "$TEMPLATECONF" ] && [ -f "$BUILDDIR/conf/templateconf.cfg" ]; then
-    TEMPLATECONF=$(cat "$BUILDDIR/conf/templateconf.cfg")
-    # The following two are no longer valid; unsetting them will automatically get them replaced
-    # with correct ones.
-    if [ "$TEMPLATECONF" = meta/conf ] || [ "$TEMPLATECONF" = meta-poky/conf ]; then
-        unset TEMPLATECONF
-        rm "$BUILDDIR/conf/templateconf.cfg"
-    fi
+TEMPLATECONF_CFG="$BUILDDIR/conf/templateconf.cfg"
+if [ -z "$TEMPLATECONF" ] && [ -f "$TEMPLATECONF_CFG" ]; then
+    TEMPLATECONF=$(cat "$TEMPLATECONF_CFG")
+    # Unset TEMPLATECONF if it is set to a known invalid value to have it

scripts/oe-setup-builddir chmod -st "$BUILDDIR/conf" 2>/dev/null || echo "WARNING: unable to chmod $BUILDD · line 50

Quoted 10 lines
scripts/oe-setup-builddir +9 -0
+    # automatically replaced with a correct one.
+    for dir in $INVALID_TEMPLATECONFS; do
+        if [ "$TEMPLATECONF" = "$dir" ]; then
+           echo "WARNING: Removing $TEMPLATECONF_CFG as it contained the invalid value '$TEMPLATECONF'"
+           unset TEMPLATECONF
+           rm "$TEMPLATECONF_CFG"
+      break;
+        fi
+    done
 fi

I've been trying to keep out this discussion but I really don't like piling in more complexity and more things we have to support because of a legacy code path we're trying to obsolete.

I also suspect this is going to be used to "subvert" the move to try and standardise and if that happens, we may as well not have bothered making changes at all :(.

Quoted 5 lines

As such I'm worried about this direction.

Cheers,

Richard

All I want is to remove the hardcoded superficial limitations and allow us that provide distros setup just as Poky is setup to continue to do so.

//Peter

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#170981): https://lists.openembedded.org/g/openembedded-core/message/170981
Mute This Topic: https://lists.openembedded.org/mt/93847437/1686489
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#5 lore raw

RE: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode invalid paths for templates

Quoted 9 lines

-----Original Message-----
From: Alexander Kanavin <[email protected]>
Sent: den 22 september 2022 17:53
To: Peter Kjellerstedt <[email protected]>
Cc: Richard Purdie <[email protected]>; openembedded- [email protected]
Subject: Re: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode invalid paths for templates

Quoted 7 lines

.templateconf is not the way to specify the location of custom templates, and never was. It got added to documentation by misunderstanding, this has now been removed, and so you need to update your tooling so that templates are set with TEMPLATECONF going forward.

Alex

[Irritated rant below, and I apologize in advance. Sorry. I know you want to improve things, but I just have to state my opinion.]

Ok, this is silly. .templateconf _is_ the documented way to specify a default TEMPLATECONF and has been so for a long time. Just because you don't use it or like it does not mean that you can just rip it out and say we now have to use some (for us) inferior method. We have 500+ developers and a huge number of CI/CD pipelines configured and built thousand times a day. They all expect the current setup, one they have used for ten years. The _exact same setup_ you get if you only fetch Poky and start a build.

All I want to do is remove hardcoded values in favor of remaining useful for all users of Yocto. Why is that a problem? I simply cannot understand it. To me, hardcoded values should never be used when a generic solution is possible. Why is it better to hardcode knowledge about Poky in OE-Core, when there is no reason to do so?

I know (or rather hope) you have some plan going forward for how to setup the build environment, but it is not there in Langdale. Also, I want to be able to continue with fetching the layers ourselves the way we always have done it and then continue to tell bitbake where to find them as we always have done without having to go through hoops or having to duplicate a lot of bitbake's internals. And I believe this was actually said during all discussions about how to add support for a common way of setting up the build environment, that it should be possible for us that already have ways to do it (be they repo, submodules, kas, whisk or whatever) to continue to do so. So please, please be mindful of simple changes that allow us to continue to use the processes we already have in place when it really doesn't cost anything.

//Peter

Quoted 98 lines
Quoted 5 lines · Peter Kjellerstedt
Quoted 5 lines

-----Original Message-----
From: Richard Purdie <[email protected]>
Sent: den 22 september 2022 15:14
To: Peter Kjellerstedt <[email protected]>; openembedded-

[email protected]

Quoted 1 line
Quoted 1 line

Subject: Re: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode

invalid paths for templates

Quoted 3 lines
Quoted 3 lines · Peter Kjellerstedt
Quoted 1 line

Previously, the paths "meta/conf" and "meta-poky/conf" were

hardcoded

Quoted 1 line
Quoted 1 line
Quoted 1 line

as invalid paths for templates. However, this is suboptimal for

other

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

distros that are setup similarly to Poky. Instead, add support for a new variable INVALID_TEMPLATECONFS, which takes a list of invalid paths. It is expected that this variable is set in the

.templateconf

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

file together with the default value for TEMPLATECONF, typically at the same time that TEMPLATECONF is updated to match the new requirements.

---
 scripts/oe-setup-builddir | 21 +++++++++++++--------
 1 file changed, 13 insertions(+), 8 deletions(-)
Quoted 5 lines
Quoted 5 lines

scripts/oe-setup-builddir chmod -st "$BUILDDIR/conf" 2>/dev/null || echo

Quoted 5 lines
scripts/oe-setup-builddir +0 -0
index 225919be92..c1148daf4f 100755
@@ -34,14 +34,19 @@ chmod -st "$BUILDDIR/conf" 2>/dev/null || echo

"WARNING: unable to chmod $BUILDD

Quoted 4 lines
Quoted 4 lines

scripts/oe-setup-builddir chmod -st "$BUILDDIR/conf" 2>/dev/null || echo "WARNING: unable to chmod $BUILDD

Quoted 4 lines
scripts/oe-setup-builddir +0 -1
 cd "$BUILDDIR" || die "Failed to change directory to $BUILDDIR!"
 
-if [ -z "$TEMPLATECONF" ] && [ -f "$BUILDDIR/conf/templateconf.cfg"

]; then

Quoted 2 lines
Quoted 2 lines

scripts/oe-setup-builddir chmod -st "$BUILDDIR/conf" 2>/dev/null || echo · line 38

Quoted 2 lines
scripts/oe-setup-builddir +0 -2
-    TEMPLATECONF=$(cat "$BUILDDIR/conf/templateconf.cfg")
-    # The following two are no longer valid; unsetting them will

automatically get them replaced

Quoted 2 lines
Quoted 2 lines

scripts/oe-setup-builddir chmod -st "$BUILDDIR/conf" 2>/dev/null || echo · line 40

Quoted 2 lines
scripts/oe-setup-builddir +0 -2
-    # with correct ones.
-    if [ "$TEMPLATECONF" = meta/conf ] || [ "$TEMPLATECONF" = meta-

poky/conf ]; then

Quoted 7 lines
Quoted 7 lines

scripts/oe-setup-builddir chmod -st "$BUILDDIR/conf" 2>/dev/null || echo "WARNING: unable to chmod $BUILDD

Quoted 7 lines
scripts/oe-setup-builddir +4 -3
-        unset TEMPLATECONF
-        rm "$BUILDDIR/conf/templateconf.cfg"
-    fi
+TEMPLATECONF_CFG="$BUILDDIR/conf/templateconf.cfg"
+if [ -z "$TEMPLATECONF" ] && [ -f "$TEMPLATECONF_CFG" ]; then
+    TEMPLATECONF=$(cat "$TEMPLATECONF_CFG")
+    # Unset TEMPLATECONF if it is set to a known invalid value to

have it

Quoted 4 lines
Quoted 4 lines

scripts/oe-setup-builddir chmod -st "$BUILDDIR/conf" 2>/dev/null || echo "WARNING: unable to chmod $BUILDD

Quoted 4 lines
scripts/oe-setup-builddir +4 -0
+    # automatically replaced with a correct one.
+    for dir in $INVALID_TEMPLATECONFS; do
+        if [ "$TEMPLATECONF" = "$dir" ]; then
+           echo "WARNING: Removing $TEMPLATECONF_CFG as it

contained the invalid value '$TEMPLATECONF'"

Quoted 9 lines
Quoted 9 lines

scripts/oe-setup-builddir chmod -st "$BUILDDIR/conf" 2>/dev/null || echo "WARNING: unable to chmod $BUILDD · line 50

Quoted 6 lines
scripts/oe-setup-builddir +5 -0
+           unset TEMPLATECONF
+           rm "$TEMPLATECONF_CFG"
+      break;
+        fi
+    done
 fi

I've been trying to keep out this discussion but I really don't like piling in more complexity and more things we have to support because

of

Quoted 16 lines
Quoted 11 lines

a legacy code path we're trying to obsolete.

I also suspect this is going to be used to "subvert" the move to try and standardise and if that happens, we may as well not have bothered making changes at all :(.

As such I'm worried about this direction.

Cheers,

Richard

All I want is to remove the hardcoded superficial limitations and allow us that provide distros setup just as Poky is setup to continue to do so.

Quoted 6 lines

//Peter

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#170981):

https://lists.openembedded.org/g/openembedded-core/message/170981

Quoted 3 lines

Mute This Topic: https://lists.openembedded.org/mt/93847437/1686489
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub

[[email protected]]

Quoted 2 lines

-=-=-=-=-=-=-=-=-=-=-=-

#6 lore raw

Re: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode invalid paths for templates

In reply to Alexander Kanavin
Quoted 2 lines · Alexander Kanavin

.templateconf is not the way to specify the location of custom templates, and never was.

That much isn't true. It was a way this has been done, it was suggested as a way people do it and it was even documented as such. The original commit does make the intent clear:

https://git.yoctoproject.org/poky/commit/.templateconf?id=614faa66a442069b8cd81ee74a01ecc5d73870d9

I think the key point is that it has been of limited use as it requires a custom repo due to the use of OEROOT. Where you had a custom repo, it was fine to use. Poky was one example.

I haven't read back through the discussions but I suspect Peter must have some kind of custom repo setup like that to be able to alter files in OEROOT.

Quoted 1 line

It got added to documentation by misunderstanding,

Again, I'm not sure about that. I think at the time, when layers were much newer things and technology was more in flux, we did think this was a valid customisation. At the time we were suggesting people might want to use tools like combo-layer to mess with repos.

What the documentation doesn't make clear is that this trick only works for repo alterations as far as I can see.

Since we don't recommend such repo alterations now and combo-layer is pretty dead, the code is of limited use and I do think it probably should be removed. I therefore don't like adding complexity to something which really shouldn't be there at all. I appreciate this does looks like it creates a challenge for Peter's situation though.

I have put off touching this particular area of code since I know there will be a ton of problems exactly like this. The success or failure of such changes does depend on how the migration is handled. A large part of this is communication and to some extend negotiation too. The fact I'm having to start trying to understand all the details is the situation I didn't really want to be in.

I suspect to get anywhere I now need to go and read all the previous threads to then try and unravel this :/.

Peter: is there a pointer you can give to the setup you're using?

Cheers,

Richard

#7 lore raw

Re: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode invalid paths for templates

First of all, I would like to understand why writing to .templateconf cannot be replaced with specifying TEMPLATECONF= in your setup.

Then we can maybe proceed to the similar question asked earlier about replacing writing to bblayer.conf with using bitbake-layers to adjust the layer list.
There was also a question about removing the step of creating a custom template altogether, and just writing a build configuration directly.

Where are the answers to those, Peter?

Alex

Quoted 55 lines · Richard Purdie
Quoted 2 lines · Alexander Kanavin

.templateconf is not the way to specify the location of custom templates, and never was.

That much isn't true. It was a way this has been done, it was suggested as a way people do it and it was even documented as such. The original commit does make the intent clear:

https://git.yoctoproject.org/poky/commit/.templateconf?id=614faa66a442069b8cd81ee74a01ecc5d73870d9

I think the key point is that it has been of limited use as it requires a custom repo due to the use of OEROOT. Where you had a custom repo, it was fine to use. Poky was one example.

I haven't read back through the discussions but I suspect Peter must have some kind of custom repo setup like that to be able to alter files in OEROOT.

Quoted 1 line

It got added to documentation by misunderstanding,

Again, I'm not sure about that. I think at the time, when layers were much newer things and technology was more in flux, we did think this was a valid customisation. At the time we were suggesting people might want to use tools like combo-layer to mess with repos.

What the documentation doesn't make clear is that this trick only works for repo alterations as far as I can see.

Since we don't recommend such repo alterations now and combo-layer is pretty dead, the code is of limited use and I do think it probably should be removed. I therefore don't like adding complexity to something which really shouldn't be there at all. I appreciate this does looks like it creates a challenge for Peter's situation though.

I have put off touching this particular area of code since I know there will be a ton of problems exactly like this. The success or failure of such changes does depend on how the migration is handled. A large part of this is communication and to some extend negotiation too. The fact I'm having to start trying to understand all the details is the situation I didn't really want to be in.

I suspect to get anywhere I now need to go and read all the previous threads to then try and unravel this :/.

Peter: is there a pointer you can give to the setup you're using?

Cheers,

Richard

A message that is not in the archive (sent off-list or not yet received): [email protected]

#9 lore raw

Re: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode invalid paths for templates

I've just written and sent a sort of 'introduction' to the official layer setup tooling:

https://lists.openembedded.org/g/openembedded-core/message/171070

Build configuration management is a separate concern, I have an introduction to that too, but not sending it yet, as particularly oe-setup-build needs to be reviewed and discussed and (hopefully) merged.

Alex

Quoted 80 lines · Alexander Kanavin via lists.openembedded.org

First of all, I would like to understand why writing to .templateconf cannot be replaced with specifying TEMPLATECONF= in your setup.

Then we can maybe proceed to the similar question asked earlier about replacing writing to bblayer.conf with using bitbake-layers to adjust the layer list.
There was also a question about removing the step of creating a custom template altogether, and just writing a build configuration directly.

Where are the answers to those, Peter?

Alex

Quoted 42 lines · Richard Purdie
Quoted 2 lines · Alexander Kanavin

.templateconf is not the way to specify the location of custom templates, and never was.

That much isn't true. It was a way this has been done, it was suggested as a way people do it and it was even documented as such. The original commit does make the intent clear:

https://git.yoctoproject.org/poky/commit/.templateconf?id=614faa66a442069b8cd81ee74a01ecc5d73870d9

I think the key point is that it has been of limited use as it requires a custom repo due to the use of OEROOT. Where you had a custom repo, it was fine to use. Poky was one example.

I haven't read back through the discussions but I suspect Peter must have some kind of custom repo setup like that to be able to alter files in OEROOT.

Quoted 1 line

It got added to documentation by misunderstanding,

Again, I'm not sure about that. I think at the time, when layers were much newer things and technology was more in flux, we did think this was a valid customisation. At the time we were suggesting people might want to use tools like combo-layer to mess with repos.

What the documentation doesn't make clear is that this trick only works for repo alterations as far as I can see.

Since we don't recommend such repo alterations now and combo-layer is pretty dead, the code is of limited use and I do think it probably should be removed. I therefore don't like adding complexity to something which really shouldn't be there at all. I appreciate this does looks like it creates a challenge for Peter's situation though.

I have put off touching this particular area of code since I know there will be a ton of problems exactly like this. The success or failure of such changes does depend on how the migration is handled. A large part of this is communication and to some extend negotiation too. The fact I'm having to start trying to understand all the details is the situation I didn't really want to be in.

Quoted 13 lines

I suspect to get anywhere I now need to go and read all the previous threads to then try and unravel this :/.

Peter: is there a pointer you can give to the setup you're using?

Cheers,

Richard

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#170995): https://lists.openembedded.org/g/openembedded-core/message/170995
Mute This Topic: https://lists.openembedded.org/mt/93847437/1686489
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#10 lore raw

RE: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode invalid paths for templates

In reply to Richard Purdie
Quoted 19 lines

-----Original Message-----
From: Richard Purdie <[email protected]>
Sent: den 22 september 2022 22:33
To: Alexander Kanavin <[email protected]>; Peter Kjellerstedt <[email protected]>
Cc: [email protected]
Subject: Re: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode invalid paths for templates

Quoted 2 lines · Alexander Kanavin

.templateconf is not the way to specify the location of custom templates, and never was.

That much isn't true. It was a way this has been done, it was suggested as a way people do it and it was even documented as such. The original commit does make the intent clear:

https://git.yoctoproject.org/poky/commit/.templateconf?id=614faa66a442069b8cd81ee74a01ecc5d73870d9

Quoted 7 lines

I think the key point is that it has been of limited use as it requires a custom repo due to the use of OEROOT. Where you had a custom repo, it was fine to use. Poky was one example.

I haven't read back through the discussions but I suspect Peter must have some kind of custom repo setup like that to be able to alter files in OEROOT.

Yes, and it is very simple to do when using repo like we do. Just add a <copyfile> line in the manifest that tells repo to copy a .templateconf file from a suitable repository and place it in the root.

Quoted 9 lines
Quoted 1 line

It got added to documentation by misunderstanding,

Again, I'm not sure about that. I think at the time, when layers were much newer things and technology was more in flux, we did think this was a valid customisation. At the time we were suggesting people might want to use tools like combo-layer to mess with repos.

What the documentation doesn't make clear is that this trick only works for repo alterations as far as I can see.

See above. At least in the case of using repo to fetch the layers, this way of specifying the default template makes perfect sense.

Quoted 5 lines

Since we don't recommend such repo alterations now and combo-layer is pretty dead, the code is of limited use and I do think it probably should be removed. I therefore don't like adding complexity to something which really shouldn't be there at all. I appreciate this does looks like it creates a challenge for Peter's situation though.

The expectation I have on Yocto is that whatever is fine and works for Poky should also be fine and work for anyone else setting up the environment in the same way.

I.e., for Poky I expect the following:

* Fetch the distribution (poky). In this case a simple git clone with the URL to the poky repository and the wanted version.
* Change to the newly fetched directory.
* Run . ./ oe-init-build-env.

Which in our case corresponds to:

* Create and change to a new directory.
* Run repo init with the URL to the manifest repository, its version and the manifest to use, followed by repo sync.
* Run . ./ oe-init-build-env.

So, as long as the layers with their correct versions have been fetched, all I should have to do is run . ./ oe-init-build-env. That is what we have always done, and I am not going to change that. Changing that would be a huge undertaking for no gain to us.

Now, the above works since I have modified our wrapper to create the template files in "templateconf/templates/axis" rather than the old "templateconf" and to create a dummy layer.conf file so the new requirements are satisfied.

The problem is that because how the templateconf.cfg file is read, its value overrides the default in the .templateconf file and thus anyone updating an existing build tree to Langdale faces an error because the old templates path is unlikely to satisfy the new requirements. This is why the code in scripts/oe-setup-builddir was changed to remove the old templateconf.cfg file if it contains a known bad path.

Since I of course do not want for my developers to face this error when they update, I have two options:

1) I can remove the old templateconf.cfg file in our wrapper. This means I have to duplicate the code in scripts/oe-buildenv-internal that calculates the BUILDDIR variable. It also means I will have to keep and maintain this code for the foreseeable future.
2) Change the code in scripts/oe-setup-builddir so that it is not hardcoded to only support OE and Poky, but instead is generic and works the same for anyone.

From my point of view, option 2 should be the natural choice.

Quoted 11 lines

I have put off touching this particular area of code since I know there will be a ton of problems exactly like this. The success or failure of such changes does depend on how the migration is handled. A large part of this is communication and to some extend negotiation too. The fact I'm having to start trying to understand all the details is the situation I didn't really want to be in.

I suspect to get anywhere I now need to go and read all the previous threads to then try and unravel this :/.

Peter: is there a pointer you can give to the setup you're using?

I gave an example in
https://lists.openembedded.org/g/openembedded-core/message/170677.
Is that what you were looking for?

Quoted 4 lines

Cheers,

Richard

//Peter

#11 lore raw

Re: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode invalid paths for templates

Quoted 12 lines · Peter Kjellerstedt

Since I of course do not want for my developers to face this error when they update, I have two options:

1) I can remove the old templateconf.cfg file in our wrapper. This means I have to duplicate the code in scripts/oe-buildenv-internal that calculates the BUILDDIR variable. It also means I will have to keep and maintain this code for the foreseeable future.
2) Change the code in scripts/oe-setup-builddir so that it is not hardcoded to only support OE and Poky, but instead is generic and works the same for anyone.

Quoted 1 line

From my point of view, option 2 should be the natural choice.

Three options. You can also instruct your developers to simply remove their build directories. Is that such a catastrophe? We do not promise that build directories stay backwards compatible and never explode on distro upgrades, we try out best, but sometimes they just don't.

Alex

A message that is not in the archive (sent off-list or not yet received): [email protected]

#13 lore raw

Re: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode invalid paths for templates

Now it's my time to rant at you, Peter. Brace yourself.

Let me just say this, you don't get to have expectations and entitlements, if none of your '500 developers' are working fulltime to help the core. I think you've been in the room last week when Nico and Philip presented the numbers and the downward trend of who is doing the work in core; what are you doing to address that situation, given the resources your organization has?

We have a small handful of exhausted maintainers, I in particular have to squeeze my maintenance work between fixing various customer BSPs, and so it is on you to help us first. Otherwise, yes, I will say this:
fix your scripts, and stop complaining. I have a large amount of manual recipe updates I need to do, because no one else wants to do it, and so I will not be arguing with you anymore.

Alex

Quoted 32 lines · Alexander Kanavin via lists.openembedded.org
Quoted 12 lines · Peter Kjellerstedt

Since I of course do not want for my developers to face this error when they update, I have two options:

1) I can remove the old templateconf.cfg file in our wrapper. This means I have to duplicate the code in scripts/oe-buildenv-internal that calculates the BUILDDIR variable. It also means I will have to keep and maintain this code for the foreseeable future.
2) Change the code in scripts/oe-setup-builddir so that it is not hardcoded to only support OE and Poky, but instead is generic and works the same for anyone.

Quoted 1 line

From my point of view, option 2 should be the natural choice.

Three options. You can also instruct your developers to simply remove their build directories. Is that such a catastrophe? We do not promise that build directories stay backwards compatible and never explode on distro upgrades, we try out best, but sometimes they just don't.

Alex

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#171107): https://lists.openembedded.org/g/openembedded-core/message/171107
Mute This Topic: https://lists.openembedded.org/mt/93847437/1686489
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#14 lore raw

RE: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode invalid paths for templates

Quoted 9 lines

-----Original Message-----
From: Alexander Kanavin <[email protected]>
Sent: den 27 september 2022 17:42
To: Peter Kjellerstedt <[email protected]>
Cc: Richard Purdie <[email protected]>; OE-core <[email protected]>
Subject: Re: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode invalid paths for templates

Quoted 8 lines

Now it's my time to rant at you, Peter. Brace yourself.

Let me just say this, you don't get to have expectations and entitlements, if none of your '500 developers' are working fulltime to help the core. I think you've been in the room last week when Nico and Philip presented the numbers and the downward trend of who is doing the work in core; what are you doing to address that situation, given the resources your organization has?

I (and the small team I am part of) is the proxy between them and Yocto. Thus I try to address their needs by making sure new versions of Yocto continues to work for them without having 500+ developers facing the same problems. I.e., it is my job to work with upstream, which is what I am trying to do here.

Quoted 3 lines

We have a small handful of exhausted maintainers, I in particular have to squeeze my maintenance work between fixing various customer BSPs, and so it is on you to help us first.

Which is exactly what I am trying to do. I am trying to make sure that what has worked, continues to work, when it is not in opposition to the actual changes that you are making. You made this exact change (that my patch addresses) for OE and Poky because you realized that there was an error that could be avoided in a simple way. And I just want to extend that to be possible to do for all distributions besides Poky. This is typically something that affect those of us who have existing distributions and not users of the new way to setup the environment.

Quoted 2 lines

Otherwise, yes, I will say this:
fix your scripts, and stop complaining.

I can (and will) of course do that, but since I believe my fix is of interest not only to us, but others who setup their environment using the .templateconf file, my primary goal is to try to improve upstream first.

Quoted 5 lines

I have a large amount of
manual recipe updates I need to do, because no one else wants to do it, and so I will not be arguing with you anymore.

Alex

//Peter

Quoted 37 lines
Quoted 22 lines · Alexander Kanavin via lists.openembedded.org
Quoted 12 lines · Peter Kjellerstedt

Since I of course do not want for my developers to face this error when they update, I have two options:

1) I can remove the old templateconf.cfg file in our wrapper. This means I have to duplicate the code in scripts/oe-buildenv-internal that calculates the BUILDDIR variable. It also means I will have to keep and maintain this code for the foreseeable future.
2) Change the code in scripts/oe-setup-builddir so that it is not hardcoded to only support OE and Poky, but instead is generic and works the same for anyone.

Quoted 1 line

From my point of view, option 2 should be the natural choice.

Three options. You can also instruct your developers to simply remove their build directories. Is that such a catastrophe? We do not promise that build directories stay backwards compatible and never explode on distro upgrades, we try out best, but sometimes they just don't.

Quoted 5 lines

Alex

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#171107):

https://lists.openembedded.org/g/openembedded-core/message/171107

Quoted 3 lines

Mute This Topic: https://lists.openembedded.org/mt/93847437/1686489
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub

[[email protected]]

Quoted 2 lines

-=-=-=-=-=-=-=-=-=-=-=-

#15 lore raw

Re: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode invalid paths for templates

Quoted 7 lines · Peter Kjellerstedt
Quoted 2 lines

Otherwise, yes, I will say this:
fix your scripts, and stop complaining.

I can (and will) of course do that, but since I believe my fix is of interest not only to us, but others who setup their environment using the .templateconf file, my primary goal is to try to improve upstream first.

Yes, but:

1. There is no evidence anyone else except Axis is actually using .templateconf.

2. .templateconf is now officially retired as a way to set the template location. The patch you've sent is completely contrary to
that: rather it makes the incorrect impression that it is still supported, and the implication is that the core upstream would still have to accommodate and think of those who are using it going forward. Yes it's mildly annoying to delete the build directory, but can you just do it and move on? And yes, make a plan for replacing it with TEMPLATECONF, which is what I've been trying to squeeze out of you for I don't know how many times.

Alex

#16 lore raw

RE: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode invalid paths for templates

Quoted 16 lines

-----Original Message-----
From: Alexander Kanavin <[email protected]>
Sent: den 27 september 2022 18:17
To: Peter Kjellerstedt <[email protected]>
Cc: Richard Purdie <[email protected]>; OE-core <[email protected]>
Subject: Re: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode invalid paths for templates

Quoted 7 lines · Peter Kjellerstedt
Quoted 2 lines

Otherwise, yes, I will say this:
fix your scripts, and stop complaining.

I can (and will) of course do that, but since I believe my fix is of interest not only to us, but others who setup their environment using the .templateconf file, my primary goal is to try to improve upstream first.

Quoted 4 lines

Yes, but:

1. There is no evidence anyone else except Axis is actually using .templateconf.

Since we obviously use our own distribution, I am limited in experience with other distributions. The one example I do know about, which I have mentioned before, is WindRiver. See line 831-835 in
https://github.com/WindRiver-Labs/wrlinux-x/blob/WRLINUX_10_21_BASE/bin/setup.py
for an example of where they do basically the same as what we do.

Quoted 6 lines

2. .templateconf is now officially retired as a way to set the template location. The patch you've sent is completely contrary to
that: rather it makes the incorrect impression that it is still supported, and the implication is that the core upstream would still have to accommodate and think of those who are using it going forward.

Which is not entirely true, is it? Support for .templateconf I still there. I am only extending what is already there to work without errors for everyone, not just OE and Poky.

Quoted 6 lines

Yes it's mildly annoying to delete the build directory, but can you just do it and move on? And yes, make a plan for replacing it with TEMPLATECONF, which is what I've been trying to squeeze out of you for I don't know how many times.

Alex

I do not understand what the problem with .templateconf is? There is a use case for being able to specify a default TEMPLATECONF, and that is what .templateconf satisfies.

There are currently three ways to specify where the templates files are (in order of preference):

1) TEMPLATECONF=
2) templateconf.cfg
3) .templateconf

In our case, the templates are always in the same location, thus we want to specify this as a default to give the developers a better user experience.

With the above three methods available, this is easy as we just set the default in the .templateconf file.

Now, if we remove support for .templateconf, how are we going to do it instead? I can of course change our wrapper around oe-build-init-env to do TEMPLATECONF=${TEMPLATECONF:-path/to/our/templates} and this might seem fine at first. However, this means that the wrapper will always set TEMPLATECONF and thus templateconf.cfg looses its meaning, which to the user means that he must now always specify TEMPLATECONF= if he wants to set it to something other than the default where before he only had to specify it the first time he initializes a new build environment.

Or I can duplicate the calculations done in the OE scripts to figure out where BUILDDIR will end up to see if there is a templateconf.cfg file available, and then use that to set TEMPLATECONF if the user hasn't specified it. All of this duplicating upstream code, which may change with any future release. (Did I mention that I hate duplicating code?)

Now I understand that you have some idea that the user should always specify TEMPLATECONF=, in which case a default TEMPLATECONF has no value. But since we have a method for fetching exactly the layers that are to be used in a given build, to me it does not make sense to also have to specify a TEMPLATECONF to match what has already been fetched. It is just duplicating information, which IMHO is bad.

What _I_ would prefer to see, is a way to specify that the build shall use the available layers. I.e., rather than a huge number of static templates configurations, I would like to see the initialization script take the layers as arguments, with the default being anything that matches meta* (or maybe more correctly, anything that has a conf/layers.conf file). As for the local.conf file, in my world, this would be made up from fragment files in the layers. So for any layer that ends up in the bblayers.conf file, the layer would be searched for any local.conf.XX files that would be concatenated together in the order of XX. This is (obviously) very similar to what we do today, so I am of course biased, but I think this model is a lot more appealing than static templates files. YMMV.

//Peter

#17 lore raw

Re: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode invalid paths for templates

Quoted 5 lines · Peter Kjellerstedt

Since we obviously use our own distribution, I am limited in experience with other distributions. The one example I do know about, which I have mentioned before, is WindRiver. See line 831-835 in
https://github.com/WindRiver-Labs/wrlinux-x/blob/WRLINUX_10_21_BASE/bin/setup.py
for an example of where they do basically the same as what we do.

Yes, they have to fix this too. And not by replacing .templateconf with TEMPLATECONF, but by writing a config directly, particularly using available tooling to compose bblayers.conf. Templates are intended for layers and version control, not as an extra intermediate step when setting up builds - the goal is to standardize build configuration management, and static templates is a starting point.

Quoted 3 lines

I do not understand what the problem with .templateconf is? There is a use case for being able to specify a default TEMPLATECONF, and that is what .templateconf satisfies.

I'd say it does not satisfy the use case, as it relies on being next to oe-init-build-env, which in turn requires custom scripting. Show how to do this using standard tools in core/poky, write a patch for the official documentation, and then you have a point.

Quoted 10 lines

There are currently three ways to specify where the templates files are (in order of preference):

1) TEMPLATECONF=
2) templateconf.cfg
3) .templateconf

In our case, the templates are always in the same location, thus we want to specify this as a default to give the developers a better user experience.

I get this, it is useful to make a pointer to a 'default' template when you have a set of layers, but the tooling for managing a modified .templateconf is absent. As things are, it is useful only as a fallback to the default core/poky template.

And if you propose such tooling, it somehow has to avoid modifying any of the layers after they have been fetched. 'repo' allows you to do this, this doesn't mean it would be an acceptable thing to do in general. I did think about it, and I don't have good or even workable ideas.

Quoted 12 lines

With the above three methods available, this is easy as we just set the default in the .templateconf file.

Now, if we remove support for .templateconf, how are we going to do it instead? I can of course change our wrapper around oe-build-init-env to do TEMPLATECONF=${TEMPLATECONF:-path/to/our/templates} and this might seem fine at first. However, this means that the wrapper will always set TEMPLATECONF and thus templateconf.cfg looses its meaning, which to the user means that he must now always specify TEMPLATECONF= if he wants to set it to something other than the default where before he only had to specify it the first time he initializes a new build environment.

I'm not sure I understand this. If the build environment has been initialized before from templates, values of TEMPLATECONF or templateconf.cfg become irrelevant, as the files in build/conf/ are already in place, and scripts will skip copying them over from templates. So you can continue to *not* have to specify TEMPLATECONF when you want to reuse an existing build directory without changing anything in it.

First time:
TEMPLATECONF=path/to/template . oe-init-build-env (will copy from template)

Second+ time:
. oe-init-build-env (will detect that no copying is needed, regardless of where the template is set to)

Can you describe the sequence of commands where the problem arises? If there is a problem here I absolutely want to see it.

Quoted 5 lines

What _I_ would prefer to see, is a way to specify that the build shall use the available layers. I.e., rather than a huge number of static templates configurations, I would like to see the initialization script take the layers as arguments, with the default being anything that matches meta* (or maybe more correctly, anything

Quoted 8 lines

that has a conf/layers.conf file). As for the local.conf file, in my world, this would be made up from fragment files in the layers. So for any layer that ends up in the bblayers.conf file, the layer would be searched for any local.conf.XX files that would be concatenated together in the order of XX. This is (obviously) very similar to what we do today, so I am of course biased, but I think this model is a lot more appealing than static templates files. YMMV.

Yes, config fragments are the next step. Whether they be automatic like you prefer, or manual ('oe-setup-build add systemd') or both is to be determined.

Alex

#18 lore raw

RE: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode invalid paths for templates

Quoted 15 lines

-----Original Message-----
From: Alexander Kanavin <[email protected]>
Sent: den 27 september 2022 21:58
To: Peter Kjellerstedt <[email protected]>
Cc: Richard Purdie <[email protected]>; OE-core <[email protected]>
Subject: Re: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode invalid paths for templates

Quoted 4 lines · Peter Kjellerstedt

Since we obviously use our own distribution, I am limited in experience with other distributions. The one example I do know about, which I have mentioned before, is WindRiver. See line 831-835 in
https://github.com/WindRiver-Labs/wrlinux-

Quoted 6 lines

x/blob/WRLINUX_10_21_BASE/bin/setup.py

Quoted 1 line

for an example of where they do basically the same as what we do.

Yes, they have to fix this too. And not by replacing .templateconf with TEMPLATECONF, but by writing a config directly, particularly using available tooling to compose bblayers.conf.

I am not too fond of the "writing the config directly" idea. The reason for that is that things change upstream (as you are well aware of, that's after all the whole reason for the discussion). This is why I prefer to go through the templates and basically only provide the list of layers to add. That way when things change upstream in the template files, we automatically follow. (And if it has not been obvious before, our wrapper uses the bblayers.conf.sample file from meta-poky and just replaces the list of layers with our list of layers. Which of course sounds like then we could just write the final config file directly, but we also want to continue to rely on oe-init-build-env for determining _where_ to write the final files.)

Quoted 5 lines

Templates are
intended for layers and version control, not as an extra intermediate step when setting up builds - the goal is to standardize build configuration management, and static templates is a starting point.

Quoted 8 lines
Quoted 3 lines

I do not understand what the problem with .templateconf is? There is a use case for being able to specify a default TEMPLATECONF, and that is what .templateconf satisfies.

I'd say it does not satisfy the use case, as it relies on being next to oe-init-build-env, which in turn requires custom scripting. Show how to do this using standard tools in core/poky, write a patch for the official documentation, and then you have a point.

Well, up until now there have been no tools in OE/poky to do anything related to setting up the layers, so I will go by the tools that I know are actually being used.

With repo and git submodules it is simple to setup the .templateconf file to match the distribution. I cannot speak for kas, as I have never used it, but I believe it uses its own way of doing things. I have tried whisk, though it was a while ago, but if I remember correctly, it also wrote the files itself. And, as of yet, I have not tried your new tools, but I plan on doing that as soon as I have the time, though I am pretty sure they do not rely on .templateconf. ;)

Quoted 8 lines
Quoted 8 lines

There are currently three ways to specify where the templates files are (in order of preference):

1) TEMPLATECONF=
2) templateconf.cfg
3) .templateconf

Quoted 8 lines
Quoted 3 lines

In our case, the templates are always in the same location, thus we want to specify this as a default to give the developers a better user experience.

I get this, it is useful to make a pointer to a 'default' template when you have a set of layers, but the tooling for managing a modified .templateconf is absent. As things are, it is useful only as a fallback to the default core/poky template.

Or anyone who sets up their distribution to mimic OE/Poky...

Quoted 31 lines

And if you propose such tooling, it somehow has to avoid modifying any of the layers after they have been fetched. 'repo' allows you to do this, this doesn't mean it would be an acceptable thing to do in general. I did think about it, and I don't have good or even workable ideas.

Quoted 12 lines

With the above three methods available, this is easy as we just set the default in the .templateconf file.

Now, if we remove support for .templateconf, how are we going to do it instead? I can of course change our wrapper around oe-build-init-env to do TEMPLATECONF=${TEMPLATECONF:-path/to/our/templates} and this might seem fine at first. However, this means that the wrapper will always set TEMPLATECONF and thus templateconf.cfg looses its meaning, which to the user means that he must now always specify TEMPLATECONF= if he wants to set it to something other than the default where before he only had to specify it the first time he initializes a new build environment.

I'm not sure I understand this. If the build environment has been initialized before from templates, values of TEMPLATECONF or templateconf.cfg become irrelevant, as the files in build/conf/ are already in place, and scripts will skip copying them over from templates. So you can continue to *not* have to specify TEMPLATECONF when you want to reuse an existing build directory without changing anything in it.

First time:
TEMPLATECONF=path/to/template . oe-init-build-env (will copy from template)

Quoted 6 lines

Second+ time:
. oe-init-build-env (will detect that no copying is needed, regardless of where the template is set to)

Can you describe the sequence of commands where the problem arises? If there is a problem here I absolutely want to see it.

Well, the real problem is that since the path _is_ stored in templateconf.cfg and this path takes precedence over the path in .templateconf, it causes problems if it is not updated, due to the new requirements on the stored path, whether it is actually used or not. :( This is what my patch tries to address.

However, having the path stored in templateconf.cfg is only really useful for anything if you delete one or more of the generated files so that they need to be regenerated. I assume that is why it was done, but I really do not know.

We use this feature in a help script we have, where you can have it update the config files by giving it an --update option, in which case it just deletes the old files in the config directory and then relies on oe-init-build-env to regenerate them. We could of course have solved this some other way, but since the feature is there, why not make use of it?

That said, having a simple way like that to update the config files is also something I would not mind seeing upstream. What we do in the local.conf is that we add a marker line at the end. Then when the user uses the --update option, we keep anything after that line but regenerate anything before it. So typically you add, e.g., MACHINE= at the end and then it is simple to update the file after an update to a new version of OE. This may not be the way local.conf is intended to be used, but I find it hard to use it if you actually uncomment any of the existing lines and then want to update the file to match a new version of OE. Thus I only use the existing lines as examples and do any modifications at the end.

But as I said, this is a convenience script that some of our users use and not part of the wrapper we have for oe-init-build-env.

Quoted 14 lines
Quoted 5 lines

What _I_ would prefer to see, is a way to specify that the build shall use the available layers. I.e., rather than a huge number of static templates configurations, I would like to see the initialization script take the layers as arguments, with the default being anything that matches meta* (or maybe more correctly, anything

Quoted 8 lines

that has a conf/layers.conf file). As for the local.conf file, in my world, this would be made up from fragment files in the layers. So for any layer that ends up in the bblayers.conf file, the layer would be searched for any local.conf.XX files that would be concatenated together in the order of XX. This is (obviously) very similar to what we do today, so I am of course biased, but I think this model is a lot more appealing than static templates files. YMMV.

Quoted 5 lines

Yes, config fragments are the next step. Whether they be automatic like you prefer, or manual ('oe-setup-build add systemd') or both is to be determined.

Alex

//Peter

#19 lore raw

Re: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode invalid paths for templates

Quoted 11 lines · Peter Kjellerstedt

I am not too fond of the "writing the config directly" idea. The reason for that is that things change upstream (as you are well aware of, that's after all the whole reason for the discussion). This is why I prefer to go through the templates and basically only provide the list of layers to add. That way when things change upstream in the template files, we automatically follow. (And if it has not been obvious before, our wrapper uses the bblayers.conf.sample file from meta-poky and just replaces the list of layers with our list of layers. Which of course sounds like then we could just write the final config file directly, but we also want to continue to rely on oe-init-build-env for determining _where_ to write the final files.)

Wait, wait, wait. If so, you can initialize using the template from poky, then you get BBPATH or BUILDDIR placed in the environment (by oe-init-build-env) telling where the config files are. So you can write to them directly (e.g. local.conf), or use 'bitbake-layers add-layer' (or 'remove-layer') to tweak the list of layers using supported tooling - after the initialization, but before any builds. No?

Alex

#20 lore raw

RE: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode invalid paths for templates

Quoted 31 lines

-----Original Message-----
From: Alexander Kanavin <[email protected]>
Sent: den 27 september 2022 23:50
To: Peter Kjellerstedt <[email protected]>
Cc: Richard Purdie <[email protected]>; OE-core <[email protected]>
Subject: Re: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode invalid paths for templates

Quoted 11 lines · Peter Kjellerstedt

I am not too fond of the "writing the config directly" idea. The reason for that is that things change upstream (as you are well aware of, that's after all the whole reason for the discussion). This is why I prefer to go through the templates and basically only provide the list of layers to add. That way when things change upstream in the template files, we automatically follow. (And if it has not been obvious before, our wrapper uses the bblayers.conf.sample file from meta-poky and just replaces the list of layers with our list of layers. Which of course sounds like then we could just write the final config file directly, but we also want to continue to rely on oe-init-build-env for determining _where_ to write the final files.)

Wait, wait, wait. If so, you can initialize using the template from poky, then you get BBPATH or BUILDDIR placed in the environment (by oe-init-build-env) telling where the config files are. So you can write to them directly (e.g. local.conf), or use 'bitbake-layers add-layer' (or 'remove-layer') to tweak the list of layers using supported tooling - after the initialization, but before any builds. No?

Quoted 1 line

Alex

I am not sure if you are suggesting that the user shall run bitbake-layers add-layer after sourcing oe-init-build-env, or if you mean that our wrapper should do it.

If it is the former, then that is obviously not how we want things to work. The layers that are fetched shall by default be part of the build. That is an absolute requirement. There is no expectancy (or want) that the user shall have to add/remove the layers manually.

If it is the latter, it wouldn't work (unless I yet again duplicate the code to figure out the build directory before sourcing the real oe-init-build-env) since before sourcing the real oe-init-build-env I cannot tell if there already is a bblayers.conf file in which case the wrapper should not add/remove any layers, and after sourcing the real oe-init-build-env it is too late as I then cannot distinguish between this being the first time oe-init-build-env is sourced or just a reinitialization of an already existing environment.

Our philosophy when we originally set this up was to keep it simple and as close to how OE/Poky works as possible. That way, if the developers have any questions and look at the Yocto documentation, the commands suggested there should work the same for us. And that is what we get from using the templates and letting the init scripts provided by OE do the work. The fact that we use the bblayers.conf.sample file from meta-poky rather than providing our own shouldn't change that. It is just a way to avoid having to manually keep our sample file in sync with what happens in OE Core.

//Peter

#21 lore raw

Re: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode invalid paths for templates

Quoted 8 lines · Peter Kjellerstedt

If it is the latter, it wouldn't work (unless I yet again duplicate the code to figure out the build directory before sourcing the real oe-init-build-env) since before sourcing the real oe-init-build-env I cannot tell if there already is a bblayers.conf file in which case the wrapper should not add/remove any layers, and after sourcing the real oe-init-build-env it is too late as I then cannot distinguish between this being the first time oe-init-build-env is sourced or just a reinitialization of an already existing environment.

Yes this is the scenario I was thinking of: what should happen inside the wrapper:

. oe-init-build-env (this takes the original poky template) bitbake-layers add-layer meta-a
bitbake-layers add-layer meta-b
...

Why would you want to distinguish between whether this is against a newly made dir
or an older one? I just checked: adding a layer that is already in bblayers.conf does nothing.

I'm also unsure why are you so adamantly against checking if the build dir already exists before the fact. It's not that difficult (either it's relative to pwd or an absolute path), and not likely to change upstream.

Alex

#22 lore raw

RE: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode invalid paths for templates

Quoted 23 lines

-----Original Message-----
From: Alexander Kanavin <[email protected]>
Sent: den 28 september 2022 13:00
To: Peter Kjellerstedt <[email protected]>
Cc: Richard Purdie <[email protected]>; OE-core <[email protected]>
Subject: Re: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode invalid paths for templates

Quoted 8 lines · Peter Kjellerstedt

If it is the latter, it wouldn't work (unless I yet again duplicate the code to figure out the build directory before sourcing the real oe-init-build-env) since before sourcing the real oe-init-build-env I cannot tell if there already is a bblayers.conf file in which case the wrapper should not add/remove any layers, and after sourcing the real oe-init-build-env it is too late as I then cannot distinguish between this being the first time oe-init-build-env is sourced or just a reinitialization of an already existing environment.

Yes this is the scenario I was thinking of: what should happen inside the wrapper:

Quoted 8 lines

. oe-init-build-env (this takes the original poky template) bitbake-layers add-layer meta-a
bitbake-layers add-layer meta-b
...

Why would you want to distinguish between whether this is against a newly made dir or an older one? I just checked: adding a layer that is already in bblayers.conf does nothing.

Because, while we want the layers fetched to be present in bblayers.conf by default, it is still expected that developers during development can add and remove layers. And if a developer has, e.g., removed a layer, it is not expected that sourcing oe-init-build-env again should restore the removed layer.

Quoted 5 lines

I'm also unsure why are you so adamantly against checking if the build dir already exists before the fact. It's not that difficult (either it's relative to pwd or an absolute path), and not likely to change upstream.

Since our wrapper is expected to behave as oe-init-build-env it must also calculate the build directory as oe-init-build-env would. And no, it is not hard to do, but it means code duplication and that we are susceptible to upstream changes. It is not a major problem and it is the way I will solve this if I cannot get a change accepted upstream, but every such small addition adds to the maintenance burden.

Quoted 1 line

Alex

//Peter

#23 lore raw

Re: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode invalid paths for templates

Quoted 5 lines · Peter Kjellerstedt

Because, while we want the layers fetched to be present in bblayers.conf by default, it is still expected that developers during development can add and remove layers. And if a developer has, e.g., removed a layer, it is not expected that sourcing oe-init-build-env again should restore the removed layer.

I wonder if we can somehow make oe-init-build-env indicate (in a way that is programmatically robust, e.g. perhaps not a marker in stdout) that it has initialized a new build directory. Ideas?

Quoted 6 lines

Since our wrapper is expected to behave as oe-init-build-env it must also calculate the build directory as oe-init-build-env would. And no, it is not hard to do, but it means code duplication and that we are susceptible to upstream changes. It is not a major problem and it is the way I will solve this if I cannot get a change accepted upstream, but every such small addition adds to the maintenance burden.

All you need is:
os.path.exists(os.path.abspath(build_dir))
(probably abspath is unneeded either, it's just there to make clear what the full path is in case it does not exist)
And no, that logic is not going to change upstream - it's far too basic and pervasive.

Tweaking the list of layers by directly manipulating the config file content when there are tools that do this for you is, I think, a worse example of duplication.

Alex

A message that is not in the archive (sent off-list or not yet received): [email protected]

#25 lore raw

Re: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode invalid paths for templates

Quoted 4 lines · Alexander Kanavin via lists.openembedded.org

I'm also unsure why are you so adamantly against checking if the build dir already exists before the fact. It's not that difficult (either it's relative to pwd or an absolute path), and not likely to change upstream.

Which is a trivial one-liner in python:
os.path.exists(os.path.abspath(build_dir))

Alex

#26 lore raw

Re: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode invalid paths for templates

In reply to Peter Kjellerstedt
Quoted 5 lines · Peter Kjellerstedt

I (and the small team I am part of) is the proxy between them and Yocto. Thus I try to address their needs by making sure new versions of Yocto continues to work for them without having 500+ developers facing the same problems. I.e., it is my job to work with upstream, which is what I am trying to do here.

Excuse me, but your job is not to 'work with upstream'. Your job is to make damn sure 'upstream' continues to exist. Which means: go to your management, describe the situation we're in (you have the numbers), and ask that they allocate at least one developer to work in upstream full time. Not drive-by patches to fix your specific product problems, but upstream proper: something that makes the project sustainable, e.g. fixing medium+/high bugs, working on new features, upgrading recipes, sending pending patches upstream. This works never ends, and no one wants to do it.

If you can do this, I promise to be more sympathetic to your particular issues, but as things stand, it's rather difficult.

Alex

#27 lore raw

RE: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode invalid paths for templates

Quoted 17 lines

-----Original Message-----
From: Alexander Kanavin <[email protected]>
Sent: den 27 september 2022 19:05
To: Peter Kjellerstedt <[email protected]>
Cc: Richard Purdie <[email protected]>; OE-core <[email protected]>
Subject: Re: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode invalid paths for templates

Quoted 5 lines · Peter Kjellerstedt

I (and the small team I am part of) is the proxy between them and Yocto. Thus I try to address their needs by making sure new versions of Yocto continues to work for them without having 500+ developers facing the same problems. I.e., it is my job to work with upstream, which is what I am trying to do here.

Quoted 3 lines

Excuse me, but your job is not to 'work with upstream'. Your job is to make damn sure 'upstream' continues to exist. Which means: go to your management, describe the situation we're in (you have the numbers),

Which is exactly why I have worked on and finally convinced my management that it is important for us to become a Yocto member, which we now are.

Quoted 2 lines

and ask that they allocate at least one developer to work in upstream full time.

That is harder. We have a hard enough time to recruit people to our tools team (which includes the build team that I am part of) as it is. For some reason, working with tools and build systems apparently is not as sexy as working with web development and AI.

Quoted 1 line

Not drive-by patches to fix your specific product problems,

I'm sorry to hear that you think all I do is drive-by patches. I thought my contributions were a little bit more appreciated than that. :(

Quoted 4 lines

but upstream proper: something that makes the project sustainable, e.g. fixing medium+/high bugs, working on new features, upgrading recipes, sending pending patches upstream. This works never ends, and no one wants to do it.

And I can assure you that everyone appreciate the ungrateful work you and others do to keep things running. We really do.

Quoted 4 lines

If you can do this, I promise to be more sympathetic to your particular issues, but as things stand, it's rather difficult.

Alex

//Peter

#28 lore raw

Re: [OE-core] [PATCH 2/2] oe-setup-builddir: Do not hardcode invalid paths for templates

Quoted 5 lines · Peter Kjellerstedt
Quoted 1 line

Not drive-by patches to fix your specific product problems,

I'm sorry to hear that you think all I do is drive-by patches. I thought my contributions were a little bit more appreciated than that. :(

They are appreciated like any contribution is, but let me clarify:
anything that is not a part of a long term commitment to maintaining some piece is a drive-by patch. If you're not sure what needs a maintainer, let me remind you of that as well:
https://git.yoctoproject.org/poky/tree/MAINTAINERS.md

Quoted 7 lines
Quoted 4 lines

but upstream proper: something that makes the project sustainable, e.g. fixing medium+/high bugs, working on new features, upgrading recipes, sending pending patches upstream. This works never ends, and no one wants to do it.

And I can assure you that everyone appreciate the ungrateful work you and others do to keep things running. We really do.

You should understand the gravity of my situation as an independent consultant, Peter. I constantly have to explain to my clients why fixing their custom internal crap (that benefits only them) is not always my top priority. Last week, I had to essentially blackmail a client that pays real money, saying that I will not do any further work for them, until they allow me to spend time upstream, and allow me to bill them for it. I am still waiting for an answer. The timing was poor as well, on the same day my friends and relatives in my home country started getting draft notices to what is likely to be useless deaths.

Alex