Skip to contentSkip to conversation

linux-yocto/stable: streamline BSP processing

PATCH In review Bruce Ashfield 3 participants 5 messages Feb 9, 2011

lore
Share & tools
lore
#0 lore raw

cover linux-yocto/stable: streamline BSP processing

Richard/Saul,

The subject is a bit deceiving, since the 'streamlining' that I've been working on for BSP / external board building for a while, actually solves a lot of the SRCREV/race conditions that we've been seeing.

With the new fetcher and some factoring of the code, I've been able to delete 99% of the anonymous python and simply the processing for new BSPs.

To create a BSP against the yocto kernels all one must do is put the variable YOCTO_KERNEL_EXTERNAL_BRANCH and indicate the branch name that should be used. The tools and processing does the rest.

I built all existing boards, and bootstrapped a BSP against the stable and devel yocto kernels while testing this. Everything worked well.

Darren: I haven't touched the preempt_rt recipes (yet), but we can streamline them as well.

This is the first step in some other refactoring and cleanups for easier inclusion of the yocto kernel parts, but we'll let this soak first before stacking more changes on top.

Pull URL: git://git.pokylinux.org/poky-contrib.git
Branch: zedd/kernel
Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=zedd/kernel

Thanks,
Bruce Ashfield <[email protected]>
---

Bruce Ashfield (1):
linux-yocto: streamline BSP bootstrapping

 meta/classes/kernel-yocto.bbclass                  |   71 +++-----------------
 .../conf/distro/include/poky-default-revisions.inc |    3 +-
 .../recipes-kernel/linux/linux-yocto-stable_git.bb |   10 ++--
 meta/recipes-kernel/linux/linux-yocto.inc          |    4 +-
 meta/recipes-kernel/linux/linux-yocto_git.bb       |    4 +
 5 files changed, 23 insertions(+), 69 deletions(-)
#1 lore raw

PATCH 1/1 linux-yocto: streamline BSP bootstrapping

In order to build BSPs that were not already integrated into the upstream linux yocto kernel AND keep the git fetcher happy, some fairly complex anonymous python sections were required.

These sections cause problems with variable expansion and SRCREV processing.

With the updated git fetcher code, we can streamline the BSP boostrapping process and drop 99% of the anonymous python code.

This commit has the following changes to support BSP boot strapping and simplication for existing BSPs.

- KMETA is set per-recipe rather than in python code
- undefined machines are no longer used, but instead common branch names are set per-recipe
- fallback machine SRCREVs are present in the default revisions file
- A new variable YOCTO_KERNEL_EXTERNAL_BRANCH should be set in the local.conf for new BSPs instead of being programatically determined in the anonymous python.
- No more explicity KMACHINE variable expansion and manipulation, since the tools and build phases no longer require it due to the per-recipe fallbacks.

Integrated/merged BSPs are unaffected by the changes and have been regression tested.

foo

---
 meta/classes/kernel-yocto.bbclass                  |   71 +++-----------------
 .../conf/distro/include/poky-default-revisions.inc |    3 +-
 .../recipes-kernel/linux/linux-yocto-stable_git.bb |   10 ++--
 meta/recipes-kernel/linux/linux-yocto.inc          |    4 +-
 meta/recipes-kernel/linux/linux-yocto_git.bb       |    4 +
 5 files changed, 23 insertions(+), 69 deletions(-)
meta/classes/kernel-yocto.bbclass +10 -61
index 4c52bca..15802fa 100644
@@ -1,61 +1,5 @@
 S = "${WORKDIR}/linux"
 
-# Determine which branch to fetch and build. Not all branches are in the
-# upstream repo (but will be locally created after the fetchers run) so 
-# a fallback branch needs to be chosen. 
-#
-# The default machine 'UNDEFINED'. If the machine is not set to a specific
-# branch in this recipe or in a recipe extension, then we fallback to a 
-# branch that is always present 'standard'. This sets the KBRANCH variable
-# and is used in the SRC_URI. The machine is then set back to ${MACHINE},
-# since futher processing will use that to create local branches
-
-SRCPV_prepend = "${@yoctokernel_variables_fixup(d)}"
-
-def yoctokernel_variables_fixup(d):
-    if d.getVar("PVFIXUPDONE", False) is "done":
-        return ""
-
-    import bb, re, string
-
-    version = bb.data.getVar("LINUX_VERSION", d, 1)
-    # 2.6.34 signifies the old-style tree, so we need some temporary
-    # conditional processing based on the kernel version.
-    if version == "2.6.34":
-        bb.data.setVar("KBRANCH", "${KMACHINE}-${LINUX_KERNEL_TYPE}", d)
-        bb.data.setVar("KMETA", "wrs_meta", d)
-        mach = bb.data.getVar("KMACHINE", d, 1)
-        if mach == "UNDEFINED":
-            bb.data.setVar("KBRANCH", "standard", d)
-            bb.data.setVar("KMACHINE", "${MACHINE}", d)
-            # track the global configuration on a bootstrapped BSP
-            bb.data.setVar("SRCREV_machine", "${SRCREV_meta}", d)
-            bb.data.setVar("BOOTSTRAP", bb.data.expand("${MACHINE}",d) + "-standard", d)
-    else:
-        # The branch for a build is:
-        #    yocto/<kernel type>/${KMACHINE} or
-        #    yocto/<kernel type>/${KMACHINE}/base
-        mach = bb.data.getVar("KMACHINE_" + bb.data.expand("${MACHINE}",d), d, 1)
-        if mach == None:
-             mach = bb.data.getVar("KMACHINE", d, 1)
-
-        bb.data.setVar("KBRANCH", mach, d)
-        bb.data.setVar("KMETA", "meta", d)
-
-        # drop the "/base" if it was on the KMACHINE
-        kmachine = mach.replace('/base','')
-        # drop everything but the last segment
-        kmachine = os.path.basename( kmachine )
-        # and then write KMACHINE back
-        bb.data.setVar('KMACHINE_' + bb.data.expand("${MACHINE}",d), kmachine, d)
-
-        if mach == "UNDEFINED":
-            bb.data.setVar('KMACHINE_' + bb.data.expand("${MACHINE}",d), bb.data.expand("${MACHINE}",d), d)
-            bb.data.setVar("KBRANCH", "yocto/standard/base", d)
-            bb.data.setVar("BOOTSTRAP", "yocto/standard/" + bb.data.expand("${MACHINE}",d), d)
-    d.setVar("PVFIXUPDONE", "done")
-    return ""
-
 do_patch() {
 	cd ${S}
 	if [ -f ${WORKDIR}/defconfig ]; then
@@ -63,12 +7,11 @@ do_patch() {
 	fi
 
 	kbranch=${KBRANCH}
-	if [ -n "${BOOTSTRAP}" ]; then
+	if [ -n "${YOCTO_KERNEL_EXTERNAL_BRANCH}" ]; then
            # switch from a generic to a specific branch
-           kbranch=${BOOTSTRAP}
+           kbranch=${YOCTO_KERNEL_EXTERNAL_BRANCH}
 	fi
 
-
 	# simply ensures that a branch of the right name has been created
 	createme ${ARCH} ${kbranch} ${defconfig}
 	if [ $? -ne 0 ]; then
@@ -131,8 +74,14 @@ addtask kernel_checkout before do_patch after do_unpack
 do_kernel_configme() {
 	echo "[INFO] doing kernel configme"
 
+	kbranch=${KBRANCH}
+	if [ -n "${YOCTO_KERNEL_EXTERNAL_BRANCH}" ]; then
+           # switch from a generic to a specific branch
+           kbranch=${YOCTO_KERNEL_EXTERNAL_BRANCH}
+	fi
+
 	cd ${S}
-	configme --reconfig --output ${B} ${KBRANCH} ${MACHINE}
+	configme --reconfig --output ${B} ${kbranch} ${MACHINE}
 	if [ $? -ne 0 ]; then
 		echo "ERROR. Could not configure ${KMACHINE}-${LINUX_KERNEL_TYPE}"
 		exit 1
@@ -165,7 +114,7 @@ do_validate_branches() {
  	target_meta_head="${SRCREV_meta}"
 
 	# nothing to do if bootstrapping
- 	if [ -n "${BOOTSTRAP}" ]; then
+ 	if [ -n "${YOCTO_KERNEL_EXTERNAL_BRANCH}" ]; then
  	 	return
  	fi
 
meta/conf/distro/include/poky-default-revisions.inc +2 -1
index 9abc4a5..a04fa9e 100644
@@ -59,7 +59,7 @@ SRCREV_pn-gypsy ??= "147"
 SRCREV_pn-inputproto ??= "7203036522ba9d4b224d282d6afc2d0b947711ee"
 SRCREV_pn-inputproto-native ??= "7203036522ba9d4b224d282d6afc2d0b947711ee"
 SRCREV_pn-inputproto-nativesdk ??= "7203036522ba9d4b224d282d6afc2d0b947711ee"
-SRCREV_pn-kern-tools-native ??= "fde7ef8fa8043607752563b9b7908f487eb2f7dd"
+SRCREV_pn-kern-tools-native ??= "4f13a46499261ea181fdc13db5459067df08aca7"
 SRCREV_pn-libdrm ??= "3f3c5be6f908272199ccf53f108b1124bfe0a00e"
 SRCREV_pn-libfakekey ??= "e8c2e412ea4a417afc1f30e32cb7bdc508b1dccc"
 SRCREV_pn-libgdbus ??= "aeab6e3c0185b271ca343b439470491b99cc587f"
@@ -110,6 +110,7 @@ SRCREV_machine_pn-linux-yocto_mpc8315e-rdb ?= "68fb0b8d73bbb336ffa4a47bc70ec1eab
 SRCREV_machine_pn-linux-yocto_beagleboard ?= "343c06b25b221aa7f7019aee83b2d976041fddce"
 SRCREV_machine_pn-linux-yocto ?= "cb4faf3af527e25feee584f39c3956ca0c9448c8"
 SRCREV_meta_pn-linux-yocto ?= "5a32d7fe3b817868ebb697d2d883d743903685ae"
+SRCREV_machine_pn-linux-yocto ?= "cb4faf3af527e25feee584f39c3956ca0c9448c8"
 SRCREV_pn-linux-libc-headers-yocto ??= "343c06b25b221aa7f7019aee83b2d976041fddce"
 SRCREV_pn-matchbox-config-gtk ??= "3ed74dfb7c57be088a5ab36e446c0ccde9fa1028"
 SRCREV_pn-matchbox-desktop-sato ??= "76"
meta/recipes-kernel/linux/linux-yocto-stable_git.bb +5 -5
index d4bee77..3c1e669 100644
@@ -1,6 +1,7 @@
 inherit kernel
 require linux-yocto.inc
 
+KMACHINE = "common_pc"
 KMACHINE_qemux86  = "common_pc"
 KMACHINE_qemux86-64  = "common_pc_64"
 KMACHINE_qemuppc  = "qemu_ppc32"
@@ -14,20 +15,19 @@ KMACHINE_beagleboard = "beagleboard"
 LINUX_VERSION ?= "2.6.34"
 LINUX_VERSION_EXTENSION ?= "-yocto-${LINUX_KERNEL_TYPE_EXTENSION}"
 
+KMETA = wrs_meta
+KBRANCH = ${KMACHINE}-${LINUX_KERNEL_TYPE_EXTENSION}
+
 PR = "r1"
 PV = "${LINUX_VERSION}+git${SRCPV}"
 SRCREV_FORMAT = "meta_machine"
 
 COMPATIBLE_MACHINE = "(qemuarm|qemux86|qemuppc|qemumips|qemux86-64|atom-pc|routerstationpro|mpc8315e-rdb|beagleboard)"
 
-# this performs a fixup on the SRCREV for new/undefined BSPs
+# this performs a fixup on historical kernel types with embedded _'s
 python __anonymous () {
     import bb, re, string
 
-    rev = bb.data.getVar("SRCREV_machine", d, 1)
-    if rev == "standard":
-        bb.data.setVar("SRCREV_machine", "${SRCREV_meta}", d)
-
     kerntype = string.replace(bb.data.expand("${LINUX_KERNEL_TYPE}", d), "_", "-")
     bb.data.setVar("LINUX_KERNEL_TYPE_EXTENSION", kerntype, d)
 }
meta/recipes-kernel/linux/linux-yocto.inc +2 -2
index 3ad849e..03935fa 100644
@@ -7,7 +7,7 @@ LIC_FILES_CHKSUM = "file://COPYING;md5=d7810fab7487fb0aad327b76f1be7cd7"
 # A KMACHINE is the mapping of a yocto $MACHINE to what is built
 # by the kernel. This is typically the branch that should be built,
 # and it can be specific to the machine or shared
-KMACHINE = "UNDEFINED"
+# KMACHINE = "UNDEFINED"
 
 # Set this to 'preempt_rt' in the local.conf if you want a real time kernel
 LINUX_KERNEL_TYPE ?= standard
@@ -19,4 +19,4 @@ addtask kernel_configme before do_configure after do_patch
 # Pick up shared functions
 inherit kernel-yocto
 
-B = "${WORKDIR}/linux-${KMACHINE}-${LINUX_KERNEL_TYPE}-build"
+B = "${WORKDIR}/linux-${MACHINE}-${LINUX_KERNEL_TYPE}-build"
meta/recipes-kernel/linux/linux-yocto_git.bb +4 -0
index cc52397..75c6fb1 100644
@@ -1,6 +1,7 @@
 inherit kernel
 require linux-yocto.inc
 
+KMACHINE = "yocto/standard/base"
 KMACHINE_qemux86  = "yocto/standard/common-pc/base"
 KMACHINE_qemux86-64  = "yocto/standard/common-pc-64/base"
 KMACHINE_qemuppc  = "yocto/standard/qemu-ppc32"
@@ -11,6 +12,9 @@ KMACHINE_routerstationpro = "yocto/standard/routerstationpro"
 KMACHINE_mpc8315e-rdb = "yocto/standard/fsl-mpc8315e-rdb"
 KMACHINE_beagleboard = "yocto/standard/beagleboard"
 
+KBRANCH = ${KMACHINE}
+KMETA = meta
+
 LINUX_VERSION ?= "2.6.37"
 LINUX_VERSION_EXTENSION ?= "-yocto-${LINUX_KERNEL_TYPE}"
 PR = "r15"
Signature · 1 line
1.7.0.4
#2 lore raw

Re: [PATCH 1/1] linux-yocto: streamline BSP bootstrapping

meta/recipes-kernel/linux/linux-yocto-stable_git.bb

Quoted 55 lines · Bruce Ashfield

In order to build BSPs that were not already integrated into the upstream linux yocto kernel AND keep the git fetcher happy, some fairly complex anonymous python sections were required.

These sections cause problems with variable expansion and SRCREV processing.

With the updated git fetcher code, we can streamline the BSP boostrapping process and drop 99% of the anonymous python code.

This commit has the following changes to support BSP boot strapping and simplication for existing BSPs.

- KMETA is set per-recipe rather than in python code
- undefined machines are no longer used, but instead common branch names are set per-recipe
- fallback machine SRCREVs are present in the default revisions file
- A new variable YOCTO_KERNEL_EXTERNAL_BRANCH should be set in the local.conf for new BSPs instead of being programatically determined in the anonymous python.
- No more explicity KMACHINE variable expansion and manipulation, since the tools and build phases no longer require it due to the per-recipe fallbacks.

Integrated/merged BSPs are unaffected by the changes and have been regression tested.

meta/conf/distro/include/poky-default-revisions.inc +2 -1
index 9abc4a5..a04fa9e 100644
@@ -59,7 +59,7 @@ SRCREV_pn-gypsy ??= "147"
 SRCREV_pn-inputproto ??= "7203036522ba9d4b224d282d6afc2d0b947711ee"
 SRCREV_pn-inputproto-native ??= "7203036522ba9d4b224d282d6afc2d0b947711ee"
 SRCREV_pn-inputproto-nativesdk ??= "7203036522ba9d4b224d282d6afc2d0b947711ee"
-SRCREV_pn-kern-tools-native ??= "fde7ef8fa8043607752563b9b7908f487eb2f7dd"
+SRCREV_pn-kern-tools-native ??= "4f13a46499261ea181fdc13db5459067df08aca7"
 SRCREV_pn-libdrm ??= "3f3c5be6f908272199ccf53f108b1124bfe0a00e"
 SRCREV_pn-libfakekey ??= "e8c2e412ea4a417afc1f30e32cb7bdc508b1dccc"
 SRCREV_pn-libgdbus ??= "aeab6e3c0185b271ca343b439470491b99cc587f"
@@ -110,6 +110,7 @@ SRCREV_machine_pn-linux-yocto_mpc8315e-rdb ?= "68fb0b8d73bbb336ffa4a47bc70ec1eab
 SRCREV_machine_pn-linux-yocto_beagleboard ?= "343c06b25b221aa7f7019aee83b2d976041fddce"
 SRCREV_machine_pn-linux-yocto ?= "cb4faf3af527e25feee584f39c3956ca0c9448c8"
 SRCREV_meta_pn-linux-yocto ?= "5a32d7fe3b817868ebb697d2d883d743903685ae"
+SRCREV_machine_pn-linux-yocto ?= "cb4faf3af527e25feee584f39c3956ca0c9448c8"
 SRCREV_pn-linux-libc-headers-yocto ??= "343c06b25b221aa7f7019aee83b2d976041fddce"
 SRCREV_pn-matchbox-config-gtk ??= "3ed74dfb7c57be088a5ab36e446c0ccde9fa1028"
 SRCREV_pn-matchbox-desktop-sato ??= "76"
meta/recipes-kernel/linux/linux-yocto-stable_git.bb +0 -0
index d4bee77..3c1e669 100644

I merged this with the exception of the second hunk above since it just adds a duplicate line.

Cheers,

Richard

#3 lore raw

Re: [PATCH 1/1] linux-yocto: streamline BSP bootstrapping

Quoted 51 lines · Richard Purdie
Quoted 50 lines · Bruce Ashfield

In order to build BSPs that were not already integrated into the upstream linux yocto kernel AND keep the git fetcher happy, some fairly complex anonymous python sections were required.

These sections cause problems with variable expansion and SRCREV processing.

With the updated git fetcher code, we can streamline the BSP boostrapping process and drop 99% of the anonymous python code.

This commit has the following changes to support BSP boot strapping and simplication for existing BSPs.

   - KMETA is set per-recipe rather than in python code
   - undefined machines are no longer used, but instead common branch names are set per-recipe
   - fallback machine SRCREVs are present in the default revisions file
   - A new variable YOCTO_KERNEL_EXTERNAL_BRANCH should be set in the local.conf for new BSPs instead of being programatically determined in the anonymous python.
   - No more explicity KMACHINE variable expansion and manipulation, since the tools and build phases no longer require it due to the per-recipe fallbacks.

Integrated/merged BSPs are unaffected by the changes and have been regression tested.

meta/conf/distro/include/poky-default-revisions.inc +0 -0
index 9abc4a5..a04fa9e 100644
@@ -59,7 +59,7 @@ SRCREV_pn-gypsy ??= "147"

 SRCREV_pn-inputproto ??= "7203036522ba9d4b224d282d6afc2d0b947711ee" SRCREV_pn-inputproto-native ??= "7203036522ba9d4b224d282d6afc2d0b947711ee" SRCREV_pn-inputproto-nativesdk ??= "7203036522ba9d4b224d282d6afc2d0b947711ee"

meta/conf/distro/include/poky-default-revisions.inc +1 -1
-SRCREV_pn-kern-tools-native ??= "fde7ef8fa8043607752563b9b7908f487eb2f7dd"
+SRCREV_pn-kern-tools-native ??= "4f13a46499261ea181fdc13db5459067df08aca7"

 SRCREV_pn-libdrm ??= "3f3c5be6f908272199ccf53f108b1124bfe0a00e" SRCREV_pn-libfakekey ??= "e8c2e412ea4a417afc1f30e32cb7bdc508b1dccc" SRCREV_pn-libgdbus ??= "aeab6e3c0185b271ca343b439470491b99cc587f"

meta/conf/distro/include/poky-default-revisions.inc +0 -0
@@ -110,6 +110,7 @@ SRCREV_machine_pn-linux-yocto_mpc8315e-rdb ?= "68fb0b8d73bbb336ffa4a47bc70ec1eab

 SRCREV_machine_pn-linux-yocto_beagleboard ?= "343c06b25b221aa7f7019aee83b2d976041fddce" SRCREV_machine_pn-linux-yocto ?= "cb4faf3af527e25feee584f39c3956ca0c9448c8" SRCREV_meta_pn-linux-yocto ?= "5a32d7fe3b817868ebb697d2d883d743903685ae" +SRCREV_machine_pn-linux-yocto ?= "cb4faf3af527e25feee584f39c3956ca0c9448c8" SRCREV_pn-linux-libc-headers-yocto ??= "343c06b25b221aa7f7019aee83b2d976041fddce" SRCREV_pn-matchbox-config-gtk ??= "3ed74dfb7c57be088a5ab36e446c0ccde9fa1028"

Quoted 8 lines

meta/recipes-kernel/linux/linux-yocto-stable_git.bb

Quoted 5 lines

 SRCREV_pn-matchbox-desktop-sato ??= "76"

meta/recipes-kernel/linux/linux-yocto-stable_git.bb +0 -0
index d4bee77..3c1e669 100644

I merged this with the exception of the second hunk above since it just adds a duplicate line.

argh. Sorry about that, I was testing with your changes in my tree and thought I had rebased them out!

Cheers,

Bruce

Quoted 10 lines

Cheers,

Richard

_______________________________________________
poky mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/poky

Signature · 2 lines
"Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end"
#4 lore raw
In reply to Bruce Ashfield

Hi Bruce,

Quoted 16 lines · Bruce Ashfield

The subject is a bit deceiving, since the 'streamlining' that I've been working on for BSP / external board building for a while, actually solves a lot of the SRCREV/race conditions that we've been seeing.

With the new fetcher and some factoring of the code, I've been able to delete 99% of the anonymous python and simply the processing for new BSPs.

To create a BSP against the yocto kernels all one must do is put the variable YOCTO_KERNEL_EXTERNAL_BRANCH and indicate the branch name that should be used. The tools and processing does the rest.

I built all existing boards, and bootstrapped a BSP against the stable and devel yocto kernels while testing this. Everything worked well.

Quoted 6 lines

Darren: I haven't touched the preempt_rt recipes (yet), but we can streamline them as well.

This is the first step in some other refactoring and cleanups for easier inclusion of the yocto kernel parts, but we'll let this soak first before stacking more changes on top.

Looks good, nice to see this cleaning up, thanks! :)

Cheers,

Richard