Skip to contentSkip to conversation

sdk: support uppercase characters in MACHINE

PATCH In review Arnout Vandecappelle (Essensium/Mind) 4 participants 5 messages Mar 15, 2022

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 sdk: support uppercase characters in MACHINE

There are no real constraints on the values that MACHINE can take, it's perfectly possible to define a machine with uppercase letters, and building for that machine works fine.

However, the (extensible) SDK targets include packages and packagegroups that have MACHINE in their name, and uppercase letters are not allowed in package names.

Therefore, convert the machine name to lowercase when it's used as a package name: for meta-environment, meta-environment-extsdk, packagegroup-cross-canadian, packagegroup-go-cross-canadian and packagegroup-rust-cross-canadian. Propagate this change to other places where those packages are referred to.

Alternatively, it would be possible to outright disallow uppercase letters in MACHINE. However, this would affect any existing configuration that currently works fine except for building the SDK.

---
I have only tested this with the base SDK, not with the extensible SDK. I also haven't tested the changes to tclibc-baremetal, tclibc-newlib, packagegroup-go-cross-canadian, packagegroup-rust-cross-canadian and maintainers.inc. packagegroup-rust-cross-canadian doesn't even seem to be used anywhere, so I guess this change would need to be propagated somewhere externally.

Ideally in addition to lowercasing, any other illegal package name variables should also be removed or converted. It's less likely though that other invalid characters would be used, since that will easily break existing recipes. Also I couldn't find a conventient function to make such a conversion.

---
 meta/classes/populate_sdk_base.bbclass                        | 2 +-
 meta/classes/populate_sdk_ext.bbclass                         | 2 +-
 meta/conf/distro/include/maintainers.inc                      | 4 ++--
 meta/conf/distro/include/tclibc-baremetal.inc                 | 2 +-
 meta/conf/distro/include/tclibc-newlib.inc                    | 2 +-
 meta/recipes-core/meta/meta-environment-extsdk.bb             | 2 +-
 meta/recipes-core/meta/meta-environment.bb                    | 2 +-
 meta/recipes-core/meta/meta-go-toolchain.bb                   | 2 +-
 .../recipes-core/packagegroups/packagegroup-cross-canadian.bb | 4 ++--
 .../packagegroups/packagegroup-go-cross-canadian.bb           | 2 +-
 .../packagegroups/packagegroup-rust-cross-canadian.bb         | 2 +-
 11 files changed, 13 insertions(+), 13 deletions(-)
meta/classes/populate_sdk_base.bbclass +1 -1
index 16f929bf59..d58c881cc3 100644
@@ -43,7 +43,7 @@ B:task-populate-sdk = "${SDK_DIR}"
 
 SDKTARGETSYSROOT = "${SDKPATH}/sysroots/${REAL_MULTIMACH_TARGET_SYS}"
 
-TOOLCHAIN_HOST_TASK ?= "nativesdk-packagegroup-sdk-host packagegroup-cross-canadian-${MACHINE}"
+TOOLCHAIN_HOST_TASK ?= "nativesdk-packagegroup-sdk-host packagegroup-cross-canadian-${@d.getVar('MACHINE', True).lower()}"
 TOOLCHAIN_HOST_TASK_ATTEMPTONLY ?= ""
 TOOLCHAIN_TARGET_TASK ?= "${@multilib_pkg_extend(d, 'packagegroup-core-standalone-sdk-target')} target-sdk-provides-dummy"
 TOOLCHAIN_TARGET_TASK_ATTEMPTONLY ?= ""
meta/classes/populate_sdk_ext.bbclass +1 -1
index e2019f9bbf..f2a2d9cde0 100644
@@ -4,7 +4,7 @@ inherit populate_sdk_base
 
 # Used to override TOOLCHAIN_HOST_TASK in the eSDK case
 TOOLCHAIN_HOST_TASK_ESDK = " \
-    meta-environment-extsdk-${MACHINE} \
+    meta-environment-extsdk-${@d.getVar('MACHINE', True).lower()} \
     "
 
 SDK_RELOCATE_AFTER_INSTALL:task-populate-sdk-ext = "0"
meta/conf/distro/include/maintainers.inc +2 -2
index a8eceaadf4..6d1b3f4ced 100644
@@ -495,8 +495,8 @@ RECIPE_MAINTAINER:pn-mesa = "Otavio Salvador <[email protected]>"
 RECIPE_MAINTAINER:pn-mesa-demos = "Otavio Salvador <[email protected]>"
 RECIPE_MAINTAINER:pn-mesa-gl = "Otavio Salvador <[email protected]>"
 RECIPE_MAINTAINER:pn-meson = "Alexander Kanavin <[email protected]>"
-RECIPE_MAINTAINER:pn-meta-environment-${MACHINE} = "Richard Purdie <[email protected]>"
-RECIPE_MAINTAINER:pn-meta-environment-extsdk-${MACHINE} = "Richard Purdie <[email protected]>"
+RECIPE_MAINTAINER:pn-meta-environment-${@d.getVar('MACHINE', True).lower()} = "Richard Purdie <[email protected]>"
+RECIPE_MAINTAINER:pn-meta-environment-extsdk-${@d.getVar('MACHINE', True).lower()} = "Richard Purdie <[email protected]>"
 RECIPE_MAINTAINER:pn-meta-extsdk-toolchain = "Richard Purdie <[email protected]>"
 RECIPE_MAINTAINER:pn-meta-go-toolchain = "Richard Purdie <[email protected]>"
 RECIPE_MAINTAINER:pn-meta-ide-support = "Richard Purdie <[email protected]>"
meta/conf/distro/include/tclibc-baremetal.inc +1 -1
index f3d27bbaae..2e216a4158 100644
@@ -27,7 +27,7 @@ BASEDEPENDS:remove:class-target = "virtual/${TARGET_PREFIX}compilerlibs"
 TARGET_OS = "elf"
 TARGET_OS:arm = "eabi"
 
-TOOLCHAIN_HOST_TASK ?= "packagegroup-cross-canadian-${MACHINE} nativesdk-qemu nativesdk-sdk-provides-dummy"
+TOOLCHAIN_HOST_TASK ?= "packagegroup-cross-canadian-${@d.getVar('MACHINE', True).lower()} nativesdk-qemu nativesdk-sdk-provides-dummy"
 TOOLCHAIN_HOST_TASK_ATTEMPTONLY ?= ""
 TOOLCHAIN_TARGET_TASK ?= "libgcc-dev"
 TOOLCHAIN_NEED_CONFIGSITE_CACHE:remove = "virtual/${MLPREFIX}libc zlib ncurses"
meta/conf/distro/include/tclibc-newlib.inc +1 -1
index 238b430e49..25c03f0aa4 100644
@@ -38,7 +38,7 @@ BASE_DEFAULT_DEPS:append:class-target = " ${NEWLIB_EXTENDED}"
 TARGET_OS = "elf"
 TARGET_OS:arm = "eabi"
 
-TOOLCHAIN_HOST_TASK ?= "packagegroup-cross-canadian-${MACHINE} nativesdk-qemu nativesdk-sdk-provides-dummy"
+TOOLCHAIN_HOST_TASK ?= "packagegroup-cross-canadian-${@d.getVar('MACHINE', True).lower()} nativesdk-qemu nativesdk-sdk-provides-dummy"
 TOOLCHAIN_TARGET_TASK ?= "${LIBC_DEPENDENCIES}"
 TOOLCHAIN_NEED_CONFIGSITE_CACHE:remove = "zlib ncurses"
 
meta/recipes-core/meta/meta-environment-extsdk.bb +1 -1
index 706312b0d6..15a86c613c 100644
@@ -2,7 +2,7 @@
 
 require meta-environment.bb
 
-PN = "meta-environment-extsdk-${MACHINE}"
+PN = "meta-environment-extsdk-${@d.getVar('MACHINE', True).lower()}"
 
 create_sdk_files:append() {
 	local sysroot=${SDKPATH}/tmp/${@os.path.relpath(d.getVar('STAGING_DIR'), d.getVar('TMPDIR'))}/${MACHINE}
meta/recipes-core/meta/meta-environment.bb +1 -1
index 7118fb2aef..146f90e7df 100644
@@ -69,7 +69,7 @@ do_install() {
     install -m 0644 -t ${D}/${SDKPATH} ${SDK_OUTPUT}/${SDKPATH}/*
 }
 
-PN = "meta-environment-${MACHINE}"
+PN = "meta-environment-${@d.getVar('MACHINE', True).lower()}"
 PACKAGES = "${PN}"
 FILES:${PN}= " \
     ${SDKPATH}/* \
meta/recipes-core/meta/meta-go-toolchain.bb +1 -1
index c24518efe3..4d405f9b94 100644
@@ -4,7 +4,7 @@ LICENSE = "MIT"
 inherit populate_sdk
 
 TOOLCHAIN_HOST_TASK:append = " \
-    packagegroup-go-cross-canadian-${MACHINE} \
+    packagegroup-go-cross-canadian-${@d.getVar('MACHINE', True).lower()} \
 "
 
 TOOLCHAIN_TARGET_TASK:append = " \
meta/recipes-core/packagegroups/packagegroup-cross-canadian.bb +2 -2
index 49c075eb11..80cc202233 100644
@@ -1,5 +1,5 @@
 SUMMARY = "Host SDK package for cross canadian toolchain"
-PN = "packagegroup-cross-canadian-${MACHINE}"
+PN = "packagegroup-cross-canadian-${@d.getVar('MACHINE', True).lower()}"
 
 inherit cross-canadian packagegroup
 
@@ -14,7 +14,7 @@ RDEPENDS:${PN} = "\
     ${@all_multilib_tune_values(d, 'BINUTILS')} \
     ${@all_multilib_tune_values(d, 'GCC')} \
     ${@all_multilib_tune_values(d, 'GDB')} \
-    meta-environment-${MACHINE} \
+    meta-environment-${@d.getVar('MACHINE', True).lower()} \
     "
 
 # When TUNE_ARCH changes but MACHINE does not (for example when a machine definition is updated), 
meta/recipes-core/packagegroups/packagegroup-go-cross-canadian.bb +1 -1
index d0596efe7a..6b10e816ba 100644
@@ -1,5 +1,5 @@
 SUMMARY = "Host SDK package for Go cross canadian toolchain"
-PN = "packagegroup-go-cross-canadian-${MACHINE}"
+PN = "packagegroup-go-cross-canadian-${@d.getVar('MACHINE', True).lower()}"
 
 inherit cross-canadian packagegroup
 
meta/recipes-core/packagegroups/packagegroup-rust-cross-canadian.bb +1 -1
index 0d4f5ec9ef..4a347cba34 100644
@@ -1,5 +1,5 @@
 SUMMARY = "Host SDK package for Rust cross canadian toolchain"
-PN = "packagegroup-rust-cross-canadian-${MACHINE}"
+PN = "packagegroup-rust-cross-canadian-${@d.getVar('MACHINE', True).lower()}"
 
 inherit cross-canadian packagegroup
 
Signature · 1 line
2.35.1
#1 lore raw
Quoted 27 lines · Arnout Vandecappelle (Essensium/Mind)

There are no real constraints on the values that MACHINE can take, it's perfectly possible to define a machine with uppercase letters, and building for that machine works fine.

However, the (extensible) SDK targets include packages and packagegroups that have MACHINE in their name, and uppercase letters are not allowed in package names.

Therefore, convert the machine name to lowercase when it's used as a package name: for meta-environment, meta-environment-extsdk, packagegroup-cross-canadian, packagegroup-go-cross-canadian and packagegroup-rust-cross-canadian. Propagate this change to other places where those packages are referred to.

Alternatively, it would be possible to outright disallow uppercase letters in MACHINE. However, this would affect any existing configuration that currently works fine except for building the SDK.

---
I have only tested this with the base SDK, not with the extensible SDK. I also haven't tested the changes to tclibc-baremetal, tclibc-newlib, packagegroup-go-cross-canadian, packagegroup-rust-cross-canadian and maintainers.inc. packagegroup-rust-cross-canadian doesn't even seem to be used anywhere, so I guess this change would need to be propagated somewhere externally.

Quoted 5 lines

Ideally in addition to lowercasing, any other illegal package name variables should also be removed or converted. It's less likely though that other invalid characters would be used, since that will easily break existing recipes. Also I couldn't find a conventient function to make such a conversion.

MACHINE is used in OVERRIDES and we only allow lower case overrides. The reasons for that are becoming more historical however the code still does require that and I'm not sure I see a pressing need to support mixed case or uppercase letters in MACHINE and complicate the code like this.

Which version of the project allows MACHINE to have uppercase characters? Are you on a very old release?

Cheers,

Richard

#2 lore raw
Quoted 34 lines · Richard Purdie
Quoted 27 lines · Arnout Vandecappelle (Essensium/Mind)

There are no real constraints on the values that MACHINE can take, it's perfectly possible to define a machine with uppercase letters, and building for that machine works fine.

However, the (extensible) SDK targets include packages and packagegroups that have MACHINE in their name, and uppercase letters are not allowed in package names.

Therefore, convert the machine name to lowercase when it's used as a package name: for meta-environment, meta-environment-extsdk, packagegroup-cross-canadian, packagegroup-go-cross-canadian and packagegroup-rust-cross-canadian. Propagate this change to other places where those packages are referred to.

Alternatively, it would be possible to outright disallow uppercase letters in MACHINE. However, this would affect any existing configuration that currently works fine except for building the SDK.

---
I have only tested this with the base SDK, not with the extensible SDK. I also haven't tested the changes to tclibc-baremetal, tclibc-newlib, packagegroup-go-cross-canadian, packagegroup-rust-cross-canadian and maintainers.inc. packagegroup-rust-cross-canadian doesn't even seem to be used anywhere, so I guess this change would need to be propagated somewhere externally.

Quoted 5 lines

Ideally in addition to lowercasing, any other illegal package name variables should also be removed or converted. It's less likely though that other invalid characters would be used, since that will easily break existing recipes. Also I couldn't find a conventient function to make such a conversion.

Quoted 7 lines

MACHINE is used in OVERRIDES and we only allow lower case overrides. The reasons for that are becoming more historical however the code still does require that and I'm not sure I see a pressing need to support mixed case or uppercase letters in MACHINE and complicate the code like this.

Which version of the project allows MACHINE to have uppercase characters? Are you on a very old release?

On dunfell. We have (received from someone else) two boards that have
uppercase characters in MACHINE. And now I notice that the other board (for which I haven't tried generating the SDK yet) even has underscores in its name, which is also illegal in PN...

I guess neither of these boards require overrides. I guess I better ask our
supplier to change the name to lowercase and without underscores.

Regards,
Arnout

#3 lore raw
In reply to Arnout Vandecappelle (Essensium/Mind)

Hi Arnout,

Quoted 14 lines · Arnout Vandecappelle (Essensium/Mind)

There are no real constraints on the values that MACHINE can take, it's perfectly possible to define a machine with uppercase letters, and building for that machine works fine.

However, the (extensible) SDK targets include packages and packagegroups that have MACHINE in their name, and uppercase letters are not allowed in package names.

Therefore, convert the machine name to lowercase when it's used as a package name: for meta-environment, meta-environment-extsdk, packagegroup-cross-canadian, packagegroup-go-cross-canadian and packagegroup-rust-cross-canadian. Propagate this change to other places where those packages are referred to.

Quoted 4 lines

Alternatively, it would be possible to outright disallow uppercase letters in MACHINE. However, this would affect any existing configuration that currently works fine except for building the SDK.

Overrides for machines starting with an uppercase letter have been broken and might still be (I was bitten by this on Thud). I don't know if this issue still applies after the migration to the newer override syntax though.

e.g. FOO_Machine will not override FOO for Machine, it'll just create a new variable named FOO_Machine instead.

Also, I seem to recall some weird issue with uppercase letter in the middle of package/recipe names. Bootlin discovered that issue in their training based on Rocko (?) a few years back because of people trying to use nInvaders as the recipe name instead of ninvaders. Pretty sure it was Alexandre who found the issue (don't remember if we ultimately found out what the issue was or just recommended not using uppercase letters at all).

Quoted 9 lines

---
I have only tested this with the base SDK, not with the extensible SDK. I also haven't tested the changes to tclibc-baremetal, tclibc-newlib, packagegroup-go-cross-canadian, packagegroup-rust-cross-canadian and maintainers.inc. packagegroup-rust-cross-canadian doesn't even seem to be used anywhere, so I guess this change would need to be propagated somewhere externally.

Quoted 5 lines

Ideally in addition to lowercasing, any other illegal package name variables should also be removed or converted. It's less likely though that other invalid characters would be used, since that will easily break existing recipes. Also I couldn't find a conventient function to make such a conversion.

I'm not sure it's worth the added complexity and confusion instead of the other way to do it which is to check for "harmful" characters in some critical strings and flat out fail the build if one is met. But I guess something needs to be done.

https://docs.yoctoproject.org/bitbake/bitbake-user-manual/bitbake-user-manual-metadata.html#conditional-metadata is the only place where I could read something about the allowed characters. I guess we should explicit this in some other places, such as MACHINE in the ref-manual.

Cheers,
Quentin

#4 lore raw
Quoted 43 lines · Quentin Schulz

Hi Arnout,

Quoted 14 lines · Arnout Vandecappelle (Essensium/Mind)

There are no real constraints on the values that MACHINE can take, it's perfectly possible to define a machine with uppercase letters, and building for that machine works fine.

However, the (extensible) SDK targets include packages and packagegroups that have MACHINE in their name, and uppercase letters are not allowed in package names.

Therefore, convert the machine name to lowercase when it's used as a package name: for meta-environment, meta-environment-extsdk, packagegroup-cross-canadian, packagegroup-go-cross-canadian and packagegroup-rust-cross-canadian. Propagate this change to other places where those packages are referred to.

Quoted 4 lines

Alternatively, it would be possible to outright disallow uppercase letters in MACHINE. However, this would affect any existing configuration that currently works fine except for building the SDK.

Overrides for machines starting with an uppercase letter have been broken and might still be (I was bitten by this on Thud). I don't know if this issue still applies after the migration to the newer override syntax though.

e.g. FOO_Machine will not override FOO for Machine, it'll just create a new variable named FOO_Machine instead.

Also, I seem to recall some weird issue with uppercase letter in the middle of package/recipe names. Bootlin discovered that issue in their training based on Rocko (?) a few years back because of people trying to use nInvaders as the recipe name instead of ninvaders. Pretty sure it was Alexandre who found the issue (don't remember if we ultimately found out what the issue was or just recommended not using uppercase letters at all).

Quoted 9 lines

---
I have only tested this with the base SDK, not with the extensible SDK. I also haven't tested the changes to tclibc-baremetal, tclibc-newlib, packagegroup-go-cross-canadian, packagegroup-rust-cross-canadian and maintainers.inc. packagegroup-rust-cross-canadian doesn't even seem to be used anywhere, so I guess this change would need to be propagated somewhere externally.

Quoted 5 lines

Ideally in addition to lowercasing, any other illegal package name variables should also be removed or converted. It's less likely though that other invalid characters would be used, since that will easily break existing recipes. Also I couldn't find a conventient function to make such a conversion.

Quoted 3 lines

I'm not sure it's worth the added complexity and confusion instead of the other way to do it which is to check for "harmful" characters in some critical strings and flat out fail the build if one is met. But I guess something needs to be done.

https://docs.yoctoproject.org/bitbake/bitbake-user-manual/bitbake-user-manual-metadata.html#conditional-metadata is the only place where I could read something about the allowed characters. I guess we should explicit this in some other places, such as MACHINE in the ref-manual.

 Ack, adding the list of allowed characters ([a-z0-9+-] I think - . is also allowed in PN but I'm not sure if it's OK for OVERRIDES) to the MACHINE documentation seems like an excellent idea.

 Even better would be a check (insane or other) that MACHINE is valid.

 Regards,
 Arnout