Skip to contentSkip to conversation

several machine specific sysroots fixes

GIT PULL Applied Dongxiao Xu 2 participants 9 messages Jan 27, 2011 – Jan 28, 2011

lore
Share & tools
lore
#0 lore raw

From: Dongxiao Xu <[email protected]>

Hi Richard,

Here are several fixes for machine specific sysroots, please help to review and pull.

The first patch is to add ${MACHINE} tag on stamps for do_populate_sysroot_setscene and do_package_setscene. The second patch is to handle the rm_work issues.
The third patch is to remove ${MACHINE} in compile and package functions in perl bb files.

Pull URL: git://git.pokylinux.org/poky-contrib.git
Branch: dxu4/distro
Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=dxu4/distro

Thanks,
Dongxiao Xu <[email protected]>
---

Dongxiao Xu (3):
bitbake: add ${MACHINE} for do_populate_sysroot_setscene and
do_package_setscene
rm_work.bbclass: handle stamp files while doing rm_work
perl: Remove ${MACHINE} dependency in compile and package.

 meta/classes/cross-canadian.bbclass       |    2 +
 meta/classes/crosssdk.bbclass             |    2 +
 meta/classes/native.bbclass               |    2 +
 meta/classes/nativesdk.bbclass            |    2 +
 meta/classes/package.bbclass              |    1 +
 meta/classes/rm_work.bbclass              |   17 +++++++++++
 meta/classes/staging.bbclass              |    1 +
 meta/recipes-devtools/perl/perl_5.12.2.bb |   42 +++++++++++++----------------
 8 files changed, 46 insertions(+), 23 deletions(-)
#1 lore raw

PATCH 1/3 bitbake: add ${MACHINE} for do_populate_sysroot_setscene and do_package_setscene

From: Dongxiao Xu <[email protected]>

${MACHINE} information is also needed to be tagged on stamps for do_populate_sysroot_setscene and do_package_setscene, otherwise if build two machines both from sstate, the second machine's sysroot will not be populated correctly.

---
 meta/classes/cross-canadian.bbclass |    2 ++
 meta/classes/crosssdk.bbclass       |    2 ++
 meta/classes/native.bbclass         |    2 ++
 meta/classes/nativesdk.bbclass      |    2 ++
 meta/classes/package.bbclass        |    1 +
 meta/classes/staging.bbclass        |    1 +
 6 files changed, 10 insertions(+), 0 deletions(-)
meta/classes/cross-canadian.bbclass +2 -0
index 993f831..db4c356 100644
@@ -82,4 +82,6 @@ export PKG_CONFIG_SYSROOT_DIR = "${STAGING_DIR_HOST}"
 SHLIBSDIR = "${STAGING_DIR}/${SDK_ARCH}-nativesdk${SDK_VENDOR}-${BUILD_OS}/shlibs"
 
 do_populate_sysroot[stamp-extra-info] = ""
+do_populate_sysroot_setscene[stamp-extra-info] = ""
 do_package[stamp-extra-info] = ""
+do_package_setscene[stamp-extra-info] = ""
meta/classes/crosssdk.bbclass +2 -0
index 75fcfef..98c0204 100644
@@ -17,4 +17,6 @@ target_prefix = "${SDKPATHNATIVE}${prefix_nativesdk}"
 target_exec_prefix = "${SDKPATHNATIVE}${exec_prefix_nativesdk}"
 
 do_populate_sysroot[stamp-extra-info] = ""
+do_populate_sysroot_setscene[stamp-extra-info] = ""
 do_package[stamp-extra-info] = ""
+do_package_setscene[stamp-extra-info] = ""
meta/classes/native.bbclass +2 -0
index 545a73e..789a163 100644
@@ -121,4 +121,6 @@ do_package_write_deb[noexec] = "1"
 do_package_write_rpm[noexec] = "1"
 
 do_populate_sysroot[stamp-extra-info] = ""
+do_populate_sysroot_setscene[stamp-extra-info] = ""
 do_package[stamp-extra-info] = ""
+do_package_setscene[stamp-extra-info] = ""
meta/classes/nativesdk.bbclass +2 -0
index 587a907..cd642c6 100644
@@ -82,4 +82,6 @@ python __anonymous () {
 }
 
 do_populate_sysroot[stamp-extra-info] = ""
+do_populate_sysroot_setscene[stamp-extra-info] = ""
 do_package[stamp-extra-info] = ""
+do_package_setscene[stamp-extra-info] = ""
meta/classes/package.bbclass +1 -0
index 1b950bb..c4e4276 100644
@@ -1095,6 +1095,7 @@ do_package[sstate-inputdirs] = "${PKGDESTWORK} ${SHLIBSWORKDIR}"
 do_package[sstate-outputdirs] = "${PKGDATA_DIR} ${SHLIBSDIR}"
 do_package[sstate-lockfile] = "${PACKAGELOCK}"
 do_package[stamp-extra-info] = "${MACHINE}"
+do_package_setscene[stamp-extra-info] = "${MACHINE}"
 do_package_setscene[dirs] = "${STAGING_DIR}"
 
 python do_package_setscene () {
meta/classes/staging.bbclass +1 -0
index 718b2f1..e2333ca 100644
@@ -86,6 +86,7 @@ do_populate_sysroot[sstate-name] = "populate-sysroot"
 do_populate_sysroot[sstate-inputdirs] = "${SYSROOT_DESTDIR}"
 do_populate_sysroot[sstate-outputdirs] = "${STAGING_DIR_HOST}/"
 do_populate_sysroot[stamp-extra-info] = "${MACHINE}"
+do_populate_sysroot_setscene[stamp-extra-info] = "${MACHINE}"
 
 python do_populate_sysroot_setscene () {
 	sstate_setscene(d)
Signature · 1 line
1.6.3.3
#2 lore raw

Re: [PATCH 1/3] bitbake: add ${MACHINE} for do_populate_sysroot_setscene and do_package_setscene

Quoted 8 lines · Dongxiao Xu

From: Dongxiao Xu <[email protected]>

${MACHINE} information is also needed to be tagged on stamps for do_populate_sysroot_setscene and do_package_setscene, otherwise if build two machines both from sstate, the second machine's sysroot will not be populated correctly.

The _setscene stuff is really an internal implementation issue within bitbake. I don't like the idea that the metadata has to get this right and I'd prefer it if bitbake always did the right thing. In this case that means looking up the extra stamp info against the parent task, not the setscene one.

I therefore merged:

http://git.pokylinux.org/cgit.cgi/poky/commit/?id=c34913ea5a4f20ce4bca3c68a65e9f64f1e3910c

which should obsolete this patch?

Good catch on finding the problem though!

Cheers,

Richard

#3 lore raw

Re: [PATCH 1/3] bitbake: add ${MACHINE} for do_populate_sysroot_setscene and do_package_setscene

Quoted 16 lines · Richard Purdie
Quoted 8 lines · Dongxiao Xu

From: Dongxiao Xu <[email protected]>

${MACHINE} information is also needed to be tagged on stamps for do_populate_sysroot_setscene and do_package_setscene, otherwise if build two machines both from sstate, the second machine's sysroot will not be populated correctly.

The _setscene stuff is really an internal implementation issue within bitbake. I don't like the idea that the metadata has to get this right and I'd prefer it if bitbake always did the right thing. In this case that means looking up the extra stamp info against the parent task, not the setscene one.

Quoted 3 lines

I therefore merged:

http://git.pokylinux.org/cgit.cgi/poky/commit/?id=c34913ea5a4f20ce4bca3c68a65e9f64f1e3910c

Yes, this approach is better then specifically handling *_setscene stamp-extra-info in meta files. Thanks!

-- Dongxiao

Quoted 8 lines

which should obsolete this patch?

Good catch on finding the problem though!

Cheers,

Richard

#4 lore raw

PATCH 2/3 rm_work.bbclass: handle stamp files while doing rm_work

In reply to Dongxiao Xu

From: Dongxiao Xu <[email protected]>

If rm_work is scheduled, we need to remove the stamps of fetch, unpack, patch, etc, while just leaving the setscene varients.

For example, we'd task stamps like:

xxx-1.4.7-r3.do_compile
xxx-1.4.7-r3.do_configure
xxx-1.4.7-r3.do_fetch
xxx-1.4.7-r3.do_generate_toolchain_file
xxx-1.4.7-r3.do_install
xxx-1.4.7-r3.do_package.emenlow
xxx-1.4.7-r3.do_package_write
xxx-1.4.7-r3.do_package_write_ipk
xxx-1.4.7-r3.do_package_write_rpm
xxx-1.4.7-r3.do_patch
xxx-1.4.7-r3.do_populate_sysroot.emenlow
xxx-1.4.7-r3.do_setscene
xxx-1.4.7-r3.do_unpack

and after rm_work, we have stamps of:

xxx-1.4.7-r3.do_package_setscene.emenlow
xxx-1.4.7-r3.do_package_write_ipk_setscene
xxx-1.4.7-r3.do_package_write_rpm_setscene
xxx-1.4.7-r3.do_populate_sysroot_setscene.emenlow

This can solve the issue we meet if we force to schedule a task after contents in ${WORKDIR} are removed.

Also this can solve the rm_work issue we met after machine specific sysroot is implemented.

Thanks for Richard's suggestion to tracking these issues down.

---
 meta/classes/rm_work.bbclass |   17 +++++++++++++++++
 1 files changed, 17 insertions(+), 0 deletions(-)
meta/classes/rm_work.bbclass +17 -0
index 260ecb0..a151f4c 100644
@@ -26,6 +26,23 @@ do_rm_work () {
     # Need to add pseudo back or subsqeuent work in this workdir
     # might fail since setscene may not rerun to recreate it
     mkdir ${WORKDIR}/pseudo/
+
+    rm -rf `ls ${STAMP}* | grep -v sigdata | grep -v do_populate_sysroot | grep -v do_package` ${STAMP}.do_package_write
+
+    for stamp in `ls ${STAMP}* | grep do_package_write_* | grep -v setscene`
+    do
+        mv $stamp $stamp\_setscene
+    done
+
+    for task in do_populate_sysroot do_package
+    do
+        if [ -e ${STAMP}.$task ]; then
+            mv ${STAMP}.$task ${STAMP}.$task\_setscene
+        elif [ -e ${STAMP}.$task.${MACHINE} ]; then
+            mv ${STAMP}.$task.${MACHINE} ${STAMP}.$task\_setscene.${MACHINE}
+        fi
+    done
+
 }
 addtask rm_work after do_${RMWORK_ORIG_TASK}
 
Signature · 1 line
1.6.3.3
#5 lore raw

Re: [PATCH 2/3] rm_work.bbclass: handle stamp files while doing rm_work

Quoted 17 lines · Dongxiao Xu
meta/classes/rm_work.bbclass +9 -0
index 260ecb0..a151f4c 100644
@@ -26,6 +26,23 @@ do_rm_work () {
     # Need to add pseudo back or subsqeuent work in this workdir
     # might fail since setscene may not rerun to recreate it
     mkdir ${WORKDIR}/pseudo/
+
+    rm -rf `ls ${STAMP}* | grep -v sigdata | grep -v do_populate_sysroot | grep -v do_package` ${STAMP}.do_package_write
+
+    for stamp in `ls ${STAMP}* | grep do_package_write_* | grep -v setscene`
+    do
+        mv $stamp $stamp\_setscene
+    done
+
+    for task in do_populate_sysroot do_package

meta/classes/rm_work.bbclass do_rm_work () { · line 47

Quoted 10 lines
meta/classes/rm_work.bbclass +8 -0
+    do
+        if [ -e ${STAMP}.$task ]; then
+            mv ${STAMP}.$task ${STAMP}.$task\_setscene
+        elif [ -e ${STAMP}.$task.${MACHINE} ]; then
+            mv ${STAMP}.$task.${MACHINE} ${STAMP}.$task\_setscene.${MACHINE}
+        fi
+    done
+
 }
 addtask rm_work after do_${RMWORK_ORIG_TASK}

This is the right approach, however, if I set BB_SIGNATURE_HANDLER = "basichash" it will break since the stamps then look like:

ncurses-5.7-r0.do_populate_sysroot.59a5b1c1ce0a1bab23be44c27ba308a5.qemux86

and also, if we add new tasks to sstate, this code would then need to be updated with the new tasknames.

I ended up writing http://git.pokylinux.org/cgit.cgi/poky/commit/?id=e8e9f56057f3732d2550375a23573ab7231eac22
which I merged into master to address the problem since its breaking the
autobuilder and people's builds out there. I'd have thought there would
be a neater way to do this but I can't find one at the moment. Any
improvements anyone can see are welcome.

Cheers,

Richard

#6 lore raw

Re: [PATCH 2/3] rm_work.bbclass: handle stamp files while doing rm_work

Quoted 34 lines · Richard Purdie
Quoted 20 lines · Dongxiao Xu
(quoted diff) +0 -0

b/meta/classes/rm_work.bbclass index 260ecb0..a151f4c 100644

meta/classes/rm_work.bbclass +0 -0
@@ -26,6 +26,23 @@ do_rm_work () {
     # Need to add pseudo back or subsqeuent work in this workdir
     # might fail since setscene may not rerun to recreate it     

mkdir ${WORKDIR}/pseudo/ +
+ rm -rf `ls ${STAMP}* | grep -v sigdata | grep -v
+ do_populate_sysroot | grep -v do_package`
${STAMP}.do_package_write + + for stamp in `ls ${STAMP}* | grep
do_package_write_* | grep -v setscene` + do + mv $stamp
$stamp\_setscene + done

meta/classes/rm_work.bbclass +7 -0
+
+    for task in do_populate_sysroot do_package
+    do
+        if [ -e ${STAMP}.$task ]; then
+            mv ${STAMP}.$task ${STAMP}.$task\_setscene
+        elif [ -e ${STAMP}.$task.${MACHINE} ]; then
+            mv ${STAMP}.$task.${MACHINE}
Quoted 4 lines

${STAMP}.$task\_setscene.${MACHINE} + fi + done
+
}
addtask rm_work after do_${RMWORK_ORIG_TASK}

This is the right approach, however, if I set BB_SIGNATURE_HANDLER = "basichash" it will break since the stamps then look like:

ncurses-5.7-r0.do_populate_sysroot.59a5b1c1ce0a1bab23be44c27ba308a5.qemux86

and also, if we add new tasks to sstate, this code would then need to be updated with the new tasknames.

Quoted 6 lines

I ended up writing
http://git.pokylinux.org/cgit.cgi/poky/commit/?id=e8e9f56057f3732d2550375a23573ab7231eac22
which I merged into master to address the problem since its breaking the autobuilder and people's builds out there. I'd have thought there would be a neater way to do this but I can't find one at the moment. Any improvements anyone can see are welcome.

Thanks for your fix, I am not aware of such stamp format, thus it's not covered in my previous fix.

Thanks,
Dongxiao

Quoted 4 lines

Cheers,

Richard

#7 lore raw

PATCH 3/3 perl: Remove ${MACHINE} dependency in compile and package.

In reply to Dongxiao Xu

From: Dongxiao Xu <[email protected]>

The judgement on ${MACHINE} is meaningless, remove it in order to avoid being rebuild on second machine.

---
 meta/recipes-devtools/perl/perl_5.12.2.bb |   42 +++++++++++++----------------
 1 files changed, 19 insertions(+), 23 deletions(-)
meta/recipes-devtools/perl/perl_5.12.2.bb +19 -23
index e7e1cb4..82801e8 100644
@@ -151,12 +151,10 @@ do_configure() {
 }
 
 do_compile() {
-        if test "${MACHINE}" != "native"; then
-            sed -i -e 's|/usr/include|${STAGING_INCDIR}|g' ext/Errno/Errno_pm.PL
-            sed -i -e 's|/usr/include|${STAGING_INCDIR}|g' cpan/Compress-Raw-Zlib/config.in
-            sed -i -e 's|/usr/lib|${STAGING_LIBDIR}|g' cpan/Compress-Raw-Zlib/config.in
+        sed -i -e 's|/usr/include|${STAGING_INCDIR}|g' ext/Errno/Errno_pm.PL
+        sed -i -e 's|/usr/include|${STAGING_INCDIR}|g' cpan/Compress-Raw-Zlib/config.in
+        sed -i -e 's|/usr/lib|${STAGING_LIBDIR}|g' cpan/Compress-Raw-Zlib/config.in
 
-        fi
         cd Cross
         oe_runmake perl LD="${CCLD}"
 }
@@ -186,24 +184,22 @@ PACKAGE_PREPROCESS_FUNCS += "perl_package_preprocess"
 
 perl_package_preprocess () {
         # Fix up installed configuration
-        if test "${MACHINE}" != "native"; then
-            sed -i -e "s,${D},,g" \
-                   -e "s,-isystem${STAGING_INCDIR} ,,g" \
-                   -e "s,${STAGING_LIBDIR},${libdir},g" \
-                   -e "s,${STAGING_BINDIR},${bindir},g" \
-                   -e "s,${STAGING_INCDIR},${includedir},g" \
-                   -e "s,${STAGING_BINDIR_NATIVE}/,,g" \
-                ${PKGD}${bindir}/h2xs \
-                ${PKGD}${bindir}/h2ph \
-                ${PKGD}${libdir}/perl/${PV}/pod/*.pod \
-                ${PKGD}${libdir}/perl/${PV}/cacheout.pl \
-                ${PKGD}${libdir}/perl/${PV}/FileCache.pm \
-                ${PKGD}${libdir}/perl/config.sh \
-                ${PKGD}${libdir}/perl/${PV}/Config.pm \
-                ${PKGD}${libdir}/perl/${PV}/Config_heavy.pl \
-                ${PKGD}${libdir}/perl/${PV}/CORE/perl.h \
-                ${PKGD}${libdir}/perl/${PV}/CORE/pp.h
-        fi
+        sed -i -e "s,${D},,g" \
+               -e "s,-isystem${STAGING_INCDIR} ,,g" \
+               -e "s,${STAGING_LIBDIR},${libdir},g" \
+               -e "s,${STAGING_BINDIR},${bindir},g" \
+               -e "s,${STAGING_INCDIR},${includedir},g" \
+               -e "s,${STAGING_BINDIR_NATIVE}/,,g" \
+            ${PKGD}${bindir}/h2xs \
+            ${PKGD}${bindir}/h2ph \
+            ${PKGD}${libdir}/perl/${PV}/pod/*.pod \
+            ${PKGD}${libdir}/perl/${PV}/cacheout.pl \
+            ${PKGD}${libdir}/perl/${PV}/FileCache.pm \
+            ${PKGD}${libdir}/perl/config.sh \
+            ${PKGD}${libdir}/perl/${PV}/Config.pm \
+            ${PKGD}${libdir}/perl/${PV}/Config_heavy.pl \
+            ${PKGD}${libdir}/perl/${PV}/CORE/perl.h \
+            ${PKGD}${libdir}/perl/${PV}/CORE/pp.h
 }
 
 PACKAGES = "perl-dbg perl perl-misc perl-dev perl-pod perl-doc perl-lib \
Signature · 1 line
1.6.3.3
#8 lore raw

Re: [PATCH 3/3] perl: Remove ${MACHINE} dependency in compile and package.

Quoted 6 lines · Dongxiao Xu

From: Dongxiao Xu <[email protected]>

The judgement on ${MACHINE} is meaningless, remove it in order to avoid being rebuild on second machine.

I've merged this into master, thanks!

Richard