linux-yocto/stable: streamline BSP processing
cover linux-yocto/stable: streamline BSP processing
Richard/Saul,
The subject is a bit deceiving, since the 'streamlining' that I've been working on for BSP / external board building for a while, actually solves a lot of the SRCREV/race conditions that we've been seeing.
With the new fetcher and some factoring of the code, I've been able to delete 99% of the anonymous python and simply the processing for new BSPs.
To create a BSP against the yocto kernels all one must do is put the variable YOCTO_KERNEL_EXTERNAL_BRANCH and indicate the branch name that should be used. The tools and processing does the rest.
I built all existing boards, and bootstrapped a BSP against the stable and devel yocto kernels while testing this. Everything worked well.
Darren: I haven't touched the preempt_rt recipes (yet), but we can streamline them as well.
This is the first step in some other refactoring and cleanups for easier inclusion of the yocto kernel parts, but we'll let this soak first before stacking more changes on top.
Pull URL: git://git.pokylinux.org/poky-contrib.git
Branch: zedd/kernel
Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=zedd/kernel
Thanks,
Bruce Ashfield <[email protected]>
---
Bruce Ashfield (1):
linux-yocto: streamline BSP bootstrapping
meta/classes/kernel-yocto.bbclass | 71 +++----------------- .../conf/distro/include/poky-default-revisions.inc | 3 +- .../recipes-kernel/linux/linux-yocto-stable_git.bb | 10 ++-- meta/recipes-kernel/linux/linux-yocto.inc | 4 +- meta/recipes-kernel/linux/linux-yocto_git.bb | 4 + 5 files changed, 23 insertions(+), 69 deletions(-)
PATCH 1/1 linux-yocto: streamline BSP bootstrapping
In order to build BSPs that were not already integrated into the upstream linux yocto kernel AND keep the git fetcher happy, some fairly complex anonymous python sections were required.
These sections cause problems with variable expansion and SRCREV processing.
With the updated git fetcher code, we can streamline the BSP boostrapping process and drop 99% of the anonymous python code.
This commit has the following changes to support BSP boot strapping and simplication for existing BSPs.
- KMETA is set per-recipe rather than in python code
- undefined machines are no longer used, but instead common branch names are set per-recipe
- fallback machine SRCREVs are present in the default revisions file
- A new variable YOCTO_KERNEL_EXTERNAL_BRANCH should be set in the local.conf for new BSPs instead of being programatically determined in the anonymous python.
- No more explicity KMACHINE variable expansion and manipulation, since the tools and build phases no longer require it due to the per-recipe fallbacks.
Integrated/merged BSPs are unaffected by the changes and have been regression tested.
foo
--- meta/classes/kernel-yocto.bbclass | 71 +++----------------- .../conf/distro/include/poky-default-revisions.inc | 3 +- .../recipes-kernel/linux/linux-yocto-stable_git.bb | 10 ++-- meta/recipes-kernel/linux/linux-yocto.inc | 4 +- meta/recipes-kernel/linux/linux-yocto_git.bb | 4 + 5 files changed, 23 insertions(+), 69 deletions(-)
meta/classes/kernel-yocto.bbclass +10 -61
@@ -1,61 +1,5 @@ S = "${WORKDIR}/linux" -# Determine which branch to fetch and build. Not all branches are in the -# upstream repo (but will be locally created after the fetchers run) so -# a fallback branch needs to be chosen. -# -# The default machine 'UNDEFINED'. If the machine is not set to a specific -# branch in this recipe or in a recipe extension, then we fallback to a -# branch that is always present 'standard'. This sets the KBRANCH variable -# and is used in the SRC_URI. The machine is then set back to ${MACHINE}, -# since futher processing will use that to create local branches - -SRCPV_prepend = "${@yoctokernel_variables_fixup(d)}" - -def yoctokernel_variables_fixup(d): - if d.getVar("PVFIXUPDONE", False) is "done": - return "" - - import bb, re, string - - version = bb.data.getVar("LINUX_VERSION", d, 1) - # 2.6.34 signifies the old-style tree, so we need some temporary - # conditional processing based on the kernel version. - if version == "2.6.34": - bb.data.setVar("KBRANCH", "${KMACHINE}-${LINUX_KERNEL_TYPE}", d) - bb.data.setVar("KMETA", "wrs_meta", d) - mach = bb.data.getVar("KMACHINE", d, 1) - if mach == "UNDEFINED": - bb.data.setVar("KBRANCH", "standard", d) - bb.data.setVar("KMACHINE", "${MACHINE}", d) - # track the global configuration on a bootstrapped BSP - bb.data.setVar("SRCREV_machine", "${SRCREV_meta}", d) - bb.data.setVar("BOOTSTRAP", bb.data.expand("${MACHINE}",d) + "-standard", d) - else: - # The branch for a build is: - # yocto/<kernel type>/${KMACHINE} or - # yocto/<kernel type>/${KMACHINE}/base - mach = bb.data.getVar("KMACHINE_" + bb.data.expand("${MACHINE}",d), d, 1) - if mach == None: - mach = bb.data.getVar("KMACHINE", d, 1) - - bb.data.setVar("KBRANCH", mach, d) - bb.data.setVar("KMETA", "meta", d) - - # drop the "/base" if it was on the KMACHINE - kmachine = mach.replace('/base','') - # drop everything but the last segment - kmachine = os.path.basename( kmachine ) - # and then write KMACHINE back - bb.data.setVar('KMACHINE_' + bb.data.expand("${MACHINE}",d), kmachine, d) - - if mach == "UNDEFINED": - bb.data.setVar('KMACHINE_' + bb.data.expand("${MACHINE}",d), bb.data.expand("${MACHINE}",d), d) - bb.data.setVar("KBRANCH", "yocto/standard/base", d) - bb.data.setVar("BOOTSTRAP", "yocto/standard/" + bb.data.expand("${MACHINE}",d), d) - d.setVar("PVFIXUPDONE", "done") - return "" - do_patch() { cd ${S} if [ -f ${WORKDIR}/defconfig ]; then
@@ -63,12 +7,11 @@ do_patch() { fi kbranch=${KBRANCH} - if [ -n "${BOOTSTRAP}" ]; then + if [ -n "${YOCTO_KERNEL_EXTERNAL_BRANCH}" ]; then # switch from a generic to a specific branch - kbranch=${BOOTSTRAP} + kbranch=${YOCTO_KERNEL_EXTERNAL_BRANCH} fi - # simply ensures that a branch of the right name has been created createme ${ARCH} ${kbranch} ${defconfig} if [ $? -ne 0 ]; then
@@ -131,8 +74,14 @@ addtask kernel_checkout before do_patch after do_unpack do_kernel_configme() { echo "[INFO] doing kernel configme" + kbranch=${KBRANCH} + if [ -n "${YOCTO_KERNEL_EXTERNAL_BRANCH}" ]; then + # switch from a generic to a specific branch + kbranch=${YOCTO_KERNEL_EXTERNAL_BRANCH} + fi + cd ${S} - configme --reconfig --output ${B} ${KBRANCH} ${MACHINE} + configme --reconfig --output ${B} ${kbranch} ${MACHINE} if [ $? -ne 0 ]; then echo "ERROR. Could not configure ${KMACHINE}-${LINUX_KERNEL_TYPE}" exit 1
@@ -165,7 +114,7 @@ do_validate_branches() { target_meta_head="${SRCREV_meta}" # nothing to do if bootstrapping - if [ -n "${BOOTSTRAP}" ]; then + if [ -n "${YOCTO_KERNEL_EXTERNAL_BRANCH}" ]; then return fi
meta/conf/distro/include/poky-default-revisions.inc +2 -1
@@ -59,7 +59,7 @@ SRCREV_pn-gypsy ??= "147" SRCREV_pn-inputproto ??= "7203036522ba9d4b224d282d6afc2d0b947711ee" SRCREV_pn-inputproto-native ??= "7203036522ba9d4b224d282d6afc2d0b947711ee" SRCREV_pn-inputproto-nativesdk ??= "7203036522ba9d4b224d282d6afc2d0b947711ee" -SRCREV_pn-kern-tools-native ??= "fde7ef8fa8043607752563b9b7908f487eb2f7dd" +SRCREV_pn-kern-tools-native ??= "4f13a46499261ea181fdc13db5459067df08aca7" SRCREV_pn-libdrm ??= "3f3c5be6f908272199ccf53f108b1124bfe0a00e" SRCREV_pn-libfakekey ??= "e8c2e412ea4a417afc1f30e32cb7bdc508b1dccc" SRCREV_pn-libgdbus ??= "aeab6e3c0185b271ca343b439470491b99cc587f"
@@ -110,6 +110,7 @@ SRCREV_machine_pn-linux-yocto_mpc8315e-rdb ?= "68fb0b8d73bbb336ffa4a47bc70ec1eab SRCREV_machine_pn-linux-yocto_beagleboard ?= "343c06b25b221aa7f7019aee83b2d976041fddce" SRCREV_machine_pn-linux-yocto ?= "cb4faf3af527e25feee584f39c3956ca0c9448c8" SRCREV_meta_pn-linux-yocto ?= "5a32d7fe3b817868ebb697d2d883d743903685ae" +SRCREV_machine_pn-linux-yocto ?= "cb4faf3af527e25feee584f39c3956ca0c9448c8" SRCREV_pn-linux-libc-headers-yocto ??= "343c06b25b221aa7f7019aee83b2d976041fddce" SRCREV_pn-matchbox-config-gtk ??= "3ed74dfb7c57be088a5ab36e446c0ccde9fa1028" SRCREV_pn-matchbox-desktop-sato ??= "76"
meta/recipes-kernel/linux/linux-yocto-stable_git.bb +5 -5
@@ -1,6 +1,7 @@ inherit kernel require linux-yocto.inc +KMACHINE = "common_pc" KMACHINE_qemux86 = "common_pc" KMACHINE_qemux86-64 = "common_pc_64" KMACHINE_qemuppc = "qemu_ppc32"
@@ -14,20 +15,19 @@ KMACHINE_beagleboard = "beagleboard" LINUX_VERSION ?= "2.6.34" LINUX_VERSION_EXTENSION ?= "-yocto-${LINUX_KERNEL_TYPE_EXTENSION}" +KMETA = wrs_meta +KBRANCH = ${KMACHINE}-${LINUX_KERNEL_TYPE_EXTENSION} + PR = "r1" PV = "${LINUX_VERSION}+git${SRCPV}" SRCREV_FORMAT = "meta_machine" COMPATIBLE_MACHINE = "(qemuarm|qemux86|qemuppc|qemumips|qemux86-64|atom-pc|routerstationpro|mpc8315e-rdb|beagleboard)" -# this performs a fixup on the SRCREV for new/undefined BSPs +# this performs a fixup on historical kernel types with embedded _'s python __anonymous () { import bb, re, string - rev = bb.data.getVar("SRCREV_machine", d, 1) - if rev == "standard": - bb.data.setVar("SRCREV_machine", "${SRCREV_meta}", d) - kerntype = string.replace(bb.data.expand("${LINUX_KERNEL_TYPE}", d), "_", "-") bb.data.setVar("LINUX_KERNEL_TYPE_EXTENSION", kerntype, d) }
meta/recipes-kernel/linux/linux-yocto.inc +2 -2
@@ -7,7 +7,7 @@ LIC_FILES_CHKSUM = "file://COPYING;md5=d7810fab7487fb0aad327b76f1be7cd7" # A KMACHINE is the mapping of a yocto $MACHINE to what is built # by the kernel. This is typically the branch that should be built, # and it can be specific to the machine or shared -KMACHINE = "UNDEFINED" +# KMACHINE = "UNDEFINED" # Set this to 'preempt_rt' in the local.conf if you want a real time kernel LINUX_KERNEL_TYPE ?= standard
@@ -19,4 +19,4 @@ addtask kernel_configme before do_configure after do_patch # Pick up shared functions inherit kernel-yocto -B = "${WORKDIR}/linux-${KMACHINE}-${LINUX_KERNEL_TYPE}-build" +B = "${WORKDIR}/linux-${MACHINE}-${LINUX_KERNEL_TYPE}-build"
meta/recipes-kernel/linux/linux-yocto_git.bb +4 -0
@@ -1,6 +1,7 @@ inherit kernel require linux-yocto.inc +KMACHINE = "yocto/standard/base" KMACHINE_qemux86 = "yocto/standard/common-pc/base" KMACHINE_qemux86-64 = "yocto/standard/common-pc-64/base" KMACHINE_qemuppc = "yocto/standard/qemu-ppc32"
@@ -11,6 +12,9 @@ KMACHINE_routerstationpro = "yocto/standard/routerstationpro" KMACHINE_mpc8315e-rdb = "yocto/standard/fsl-mpc8315e-rdb" KMACHINE_beagleboard = "yocto/standard/beagleboard" +KBRANCH = ${KMACHINE} +KMETA = meta + LINUX_VERSION ?= "2.6.37" LINUX_VERSION_EXTENSION ?= "-yocto-${LINUX_KERNEL_TYPE}" PR = "r15"
Signature · 1 line
1.7.0.4
Hi Bruce,
Quoted 16 lines · Bruce Ashfield
The subject is a bit deceiving, since the 'streamlining' that I've been working on for BSP / external board building for a while, actually solves a lot of the SRCREV/race conditions that we've been seeing.
With the new fetcher and some factoring of the code, I've been able to delete 99% of the anonymous python and simply the processing for new BSPs.
To create a BSP against the yocto kernels all one must do is put the variable
YOCTO_KERNEL_EXTERNAL_BRANCHand indicate the branch name that should be used. The tools and processing does the rest.I built all existing boards, and bootstrapped a BSP against the stable and devel yocto kernels while testing this. Everything worked well.
Quoted 6 lines
Darren: I haven't touched the
preempt_rtrecipes (yet), but we can streamline them as well.This is the first step in some other refactoring and cleanups for easier inclusion of the yocto kernel parts, but we'll let this soak first before stacking more changes on top.
Looks good, nice to see this cleaning up, thanks! :)
Cheers,
Richard
Re: [PATCH 1/1] linux-yocto: streamline BSP bootstrapping
In reply to Bruce Ashfieldmeta/recipes-kernel/linux/linux-yocto-stable_git.bb
Quoted 55 lines · Bruce Ashfield
In order to build BSPs that were not already integrated into the upstream linux yocto kernel AND keep the git fetcher happy, some fairly complex anonymous python sections were required.
These sections cause problems with variable expansion and SRCREV processing.
With the updated git fetcher code, we can streamline the BSP boostrapping process and drop 99% of the anonymous python code.
This commit has the following changes to support BSP boot strapping and simplication for existing BSPs.
- KMETA is set per-recipe rather than in python code
- undefined machines are no longer used, but instead common branch names are set per-recipe
- fallback machine SRCREVs are present in the default revisions file
- A new variableYOCTO_KERNEL_EXTERNAL_BRANCHshould be set in the local.conf for new BSPs instead of being programatically determined in the anonymous python.
- No more explicity KMACHINE variable expansion and manipulation, since the tools and build phases no longer require it due to the per-recipe fallbacks.Integrated/merged BSPs are unaffected by the changes and have been regression tested.
Signed-off-by Bruce Ashfield <[email protected]>meta/conf/distro/include/poky-default-revisions.inc +2
-1@@ -59,7 +59,7 @@ SRCREV_pn-gypsy ??= "147" SRCREV_pn-inputproto ??= "7203036522ba9d4b224d282d6afc2d0b947711ee" SRCREV_pn-inputproto-native ??= "7203036522ba9d4b224d282d6afc2d0b947711ee" SRCREV_pn-inputproto-nativesdk ??= "7203036522ba9d4b224d282d6afc2d0b947711ee" -SRCREV_pn-kern-tools-native ??= "fde7ef8fa8043607752563b9b7908f487eb2f7dd" +SRCREV_pn-kern-tools-native ??= "4f13a46499261ea181fdc13db5459067df08aca7" SRCREV_pn-libdrm ??= "3f3c5be6f908272199ccf53f108b1124bfe0a00e" SRCREV_pn-libfakekey ??= "e8c2e412ea4a417afc1f30e32cb7bdc508b1dccc" SRCREV_pn-libgdbus ??= "aeab6e3c0185b271ca343b439470491b99cc587f"@@ -110,6 +110,7 @@ SRCREV_machine_pn-linux-yocto_mpc8315e-rdb ?= "68fb0b8d73bbb336ffa4a47bc70ec1eab SRCREV_machine_pn-linux-yocto_beagleboard ?= "343c06b25b221aa7f7019aee83b2d976041fddce" SRCREV_machine_pn-linux-yocto ?= "cb4faf3af527e25feee584f39c3956ca0c9448c8" SRCREV_meta_pn-linux-yocto ?= "5a32d7fe3b817868ebb697d2d883d743903685ae" +SRCREV_machine_pn-linux-yocto ?= "cb4faf3af527e25feee584f39c3956ca0c9448c8" SRCREV_pn-linux-libc-headers-yocto ??= "343c06b25b221aa7f7019aee83b2d976041fddce" SRCREV_pn-matchbox-config-gtk ??= "3ed74dfb7c57be088a5ab36e446c0ccde9fa1028" SRCREV_pn-matchbox-desktop-sato ??= "76"meta/recipes-kernel/linux/linux-yocto-stable_git.bb +0
-0
I merged this with the exception of the second hunk above since it just adds a duplicate line.
Cheers,
Richard
Re: [PATCH 1/1] linux-yocto: streamline BSP bootstrapping
Quoted 51 lines · Richard Purdie
Quoted 50 lines · Bruce Ashfield
In order to build BSPs that were not already integrated into the upstream linux yocto kernel AND keep the git fetcher happy, some fairly complex anonymous python sections were required.
These sections cause problems with variable expansion and SRCREV processing.
With the updated git fetcher code, we can streamline the BSP boostrapping process and drop 99% of the anonymous python code.
This commit has the following changes to support BSP boot strapping and simplication for existing BSPs.
- KMETA is set per-recipe rather than in python code
- undefined machines are no longer used, but instead common branch names are set per-recipe
- fallback machine SRCREVs are present in the default revisions file
- A new variableYOCTO_KERNEL_EXTERNAL_BRANCHshould be set in the local.conf for new BSPs instead of being programatically determined in the anonymous python.
- No more explicity KMACHINE variable expansion and manipulation, since the tools and build phases no longer require it due to the per-recipe fallbacks.Integrated/merged BSPs are unaffected by the changes and have been regression tested.
Signed-off-by Bruce Ashfield <[email protected]>meta/conf/distro/include/poky-default-revisions.inc +0
-0@@ -59,7 +59,7 @@ SRCREV_pn-gypsy ??= "147"SRCREV_pn-inputproto ??= "
7203036522ba9d4b224d282d6afc2d0b947711ee" SRCREV_pn-inputproto-native ??= "7203036522ba9d4b224d282d6afc2d0b947711ee" SRCREV_pn-inputproto-nativesdk ??= "7203036522ba9d4b224d282d6afc2d0b947711ee"meta/conf/distro/include/poky-default-revisions.inc +1
-1-SRCREV_pn-kern-tools-native ??= "fde7ef8fa8043607752563b9b7908f487eb2f7dd" +SRCREV_pn-kern-tools-native ??= "4f13a46499261ea181fdc13db5459067df08aca7"SRCREV_pn-libdrm ??= "
3f3c5be6f908272199ccf53f108b1124bfe0a00e" SRCREV_pn-libfakekey ??= "e8c2e412ea4a417afc1f30e32cb7bdc508b1dccc" SRCREV_pn-libgdbus ??= "aeab6e3c0185b271ca343b439470491b99cc587f"meta/conf/distro/include/poky-default-revisions.inc +0
-0@@ -110,6 +110,7 @@ SRCREV_machine_pn-linux-yocto_mpc8315e-rdb ?= "68fb0b8d73bbb336ffa4a47bc70ec1eabSRCREV_machine_pn-linux-yocto_beagleboard ?= "
343c06b25b221aa7f7019aee83b2d976041fddce" SRCREV_machine_pn-linux-yocto ?= "cb4faf3af527e25feee584f39c3956ca0c9448c8" SRCREV_meta_pn-linux-yocto ?= "5a32d7fe3b817868ebb697d2d883d743903685ae" +SRCREV_machine_pn-linux-yocto ?= "cb4faf3af527e25feee584f39c3956ca0c9448c8" SRCREV_pn-linux-libc-headers-yocto ??= "343c06b25b221aa7f7019aee83b2d976041fddce" SRCREV_pn-matchbox-config-gtk ??= "3ed74dfb7c57be088a5ab36e446c0ccde9fa1028"
Quoted 8 lines
meta/recipes-kernel/linux/linux-yocto-stable_git.bb
Quoted 5 lines
SRCREV_pn-matchbox-desktop-sato ??= "76"
meta/recipes-kernel/linux/linux-yocto-stable_git.bb +0
-0I merged this with the exception of the second hunk above since it just adds a duplicate line.
argh. Sorry about that, I was testing with your changes in my tree and thought I had rebased them out!
Cheers,
Bruce
Quoted 10 lines
Cheers,
Richard
_______________________________________________
poky mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/poky
Signature · 2 lines
"Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end"