gcc support shared source for basichash

This is part of the Yocto Bug #1555 fixing.

We use ".done" file along with the task lockfile to allow multiple gcc-related recipes sharing the same source directory when basichash is used as the default signature handler.

The following changes since commit bd047935305c872b565f30b46c94b7077e5fb3a2:
Richard Purdie (1):
patch.bbclass: Ensure the DATE and SRCDATE variable exclusions apply to the correct function

are available in the git repository at:

git://git.yoctoproject.org/poky-contrib llu/bug1555-gcc
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=llu/bug1555-gcc

Lianhao Lu (1):
gcc-common.inc: Added shared source support for basichash.

 meta/recipes-devtools/gcc/gcc-common.inc |   56 ++++++++++++++++++++++++++++++
 1 files changed, 56 insertions(+), 0 deletions(-)

[YOCTO #1555]
Adding new mechanism to allow multiple gcc related recipes sharing the same src directory when using the basichash as the signature handler.

Using the same stamp file for multiple gcc related recipes to share the same src directory is no longer working when using the basichash signature.

---
 meta/recipes-devtools/gcc/gcc-common.inc |   56 ++++++++++++++++++++++++++++++
 1 files changed, 56 insertions(+), 0 deletions(-)

index fe112d9..4f8e18f 100644
@@ -63,6 +63,62 @@ do_unpack[stamp-base] = "${SS}"
 do_headerfix[stamp-base] = "${SS}"
 do_patch[stamp-base] = "${SS}"
 
+# Shared source for basichash
+def check_shared_taskdone(d, task):
+    import os.path, bb.utils
+    basedir = d.getVar('SW', True)
+    bb.utils.mkdirhier(basedir)
+    fn = os.path.join(basedir, "%s.done" % (task))
+    return os.path.exists(fn)
+
+def mark_shared_taskdone(d, task):
+    import os.path, bb.utils
+    basedir = d.getVar('SW', True)
+    bb.utils.mkdirhier(basedir)
+    fn = os.path.join(basedir, "%s.done" % (task))
+    f = open(fn, "w")
+    f.close()
+
+do_fetch[lockfiles] = "${SW}.fetch.lock"
+do_unpack[lockfiles] = "${SW}.unpack.lock"
+do_headerfix[lockfiles] = "${SW}.headerfix.lock"
+do_patch[lockfiles] = "${SW}.patch.lock"
+
+python do_fetch_prepend() {
+	if check_shared_taskdone(d, 'fetch'):
+		return
+}
+python do_fetch_append() {
+	mark_shared_taskdone(d, 'fetch')
+}
+
+python do_unpack_prepend() {
+	if check_shared_taskdone(d, 'unpack'):
+		return
+}
+python do_unpack_append() {
+	mark_shared_taskdone(d, 'unpack')
+}
+
+python do_patch_prepend() {
+	if check_shared_taskdone(d, 'patch'):
+		return
+}
+python do_patch_append() {
+	mark_shared_taskdone(d, 'patch')
+}
+
+do_headerfix_prepend() {
+	if [ -e "${SW}/headerfix.done" ]; then
+		return 0
+	fi
+}
+
+do_headerfix_append() {
+	mkdir -p "${SW}"
+	touch "${SW}/headerfix.done"
+}
+
 # SW means Shared Work directory
 SW = "${TMPDIR}/work-shared/gcc-${PV}-${PR}"
 WORKDIR_task-unpack = "${SW}"

1.7.0.4

Can you please check if this can be fixed by:

index d014980..3a82720 100644
@@ -77,6 +77,9 @@ do_headerfix () {
 
 addtask headerfix after do_unpack before do_patch
 
+CROSS_TARGET_SYS_DIR[vardepsexclude] = "PN"
+CROSS_TARGET_SYS_DIR[vardepvalue] = "1"
+
 do_configure_prepend () {
        # teach gcc to find correct target includedir when checking libc ssp support
        mkdir -p ${B}/gcc

as discussed in the "sstate info" thread?

Cheers,

Richard

Richard Purdie wrote on 2012-01-17:

By using this patch, gcc-cross-initial, gcc-cross-intermediate, gcc-cross would have the same signatures for tasks like do_fetch, do_unpack, do_headerfix, do_patch. But libgcc still has different signature with gcc-cross-initial for task do_patch.

By bitbake-diffsigs the do_patch signature files between gcc-cross-initial and libgcc, it says:
Dependency on task gcc-cross-initial_4.6.bb.do_unpack was added
Dependency on task gcc-cross-initial_4.6.bb.do_headerfix was added
Dependency on task libgcc_4.6.bb.do_unpack was removed

Besides, gcc-runtime seems have an additional patch file "fortran-cross-compile-hack.patch" which was never got be used. This would results all the signature difference between gcc-cross-initial and gcc-runtime. After I remove that unused patch file, the signatures begin to differ in task do_headerfix and do_patch. By bitbake-diffsigs the do_headerfix sigangure files between gcc-cross-initial and gcc-runtime, we could see tons of variable values, variable dependencies, task dependencies difference.

So I think this patch alone is not enough to solve all the problems here.

Best Regards,
Lianhao

Agreed, there are further issues. The patch should really move into the main SRC_URI which would resolve that problem. libgcc should really be able to see the do_headerfix task so we could move that from gcc-configure-common to gcc-common.inc.

This might not be all of the differences but should get us closer to solving this problem!

Cheers,

Richard

Richard Purdie wrote on 2012-01-19:

By moving the gcc-runtime specific patch into main SRC_URI, gcc-runtime still has different signature with gcc-cross-initial for task do_headerfix. Bitbake-diffsigs gives the following information. Is it possible that the difference comes from the "inherit cross" in gcc-cross.inc?

$ bitbake-diffsigs gcc-cross-initial-4.6.2+svnr181430-r20.do_headerfix.sigdata.6305744411a4f2d0055d98b5d8c2dd51 gcc-runtime-4.6.2+svnr181430-r20.do_headerfix.sigdata.2d8cad118dd54b302eadae3db11021fa
Task dependencies changed from:
['ABIEXTENSION', 'AR', 'ARMPKGARCH', 'ARMPKGSFX_DSP', 'ARMPKGSFX_EABI', 'ARMPKGSFX_ENDIAN', 'ARMPKGSFX_FPU', 'ARMPKGSFX_THUMB', 'AS', 'ASNEEDED', 'BRANCH', 'BUILD_AR', 'BUILD_ARCH', 'BUILD_AS', 'BUILD_AS_ARCH', 'BUILD_CC', 'BUILD_CCLD', 'BUILD_CC_ARCH', 'BUILD_CFLAGS', 'BUILD_CPP', 'BUILD_CPPFLAGS', 'BUILD_CXX', 'BUILD_CXXFLAGS', 'BUILD_F77', 'BUILD_LD', 'BUILD_LDFLAGS', 'BUILD_LD_ARCH', 'BUILD_NM', 'BUILD_OPTIMIZATION', 'BUILD_OS', 'BUILD_PREFIX', 'BUILD_RANLIB', 'BUILD_STRIP', 'BUILD_SYS', 'BUILD_VENDOR', 'CC', 'CCACHE', 'CCACHE_DIR', 'CCLD', 'CFLAGS', 'CONFIG_SITE', 'CPP', 'CPPFLAGS', 'CROSS_TARGET_SYS_DIR', 'CXX', 'CXXFLAGS', 'DEBUG_BUILD', 'DEBUG_FLAGS', 'DEBUG_OPTIMIZATION', 'F77', 'FULL_OPTIMIZATION', 'HOST_ARCH', 'HOST_AS_ARCH', 'HOST_CC_ARCH', 'HOST_LD_ARCH', 'HOST_OS', 'HOST_PREFIX', 'HOST_VENDOR', 'LC_ALL', 'LD', 'LDFLAGS', 'LIBCEXTENSION', 'LINKER_HASH_STYLE', 'MAKE', 'MULTIMACH_HOST_SYS', 'NM', 'OBJCOPY', 'OBJDUMP', 'OPIEDIR', 'PACKAGE_ARCH', 'PATCH_GET', 'PKG_CONFIG_DIR', 'PKG_CONFIG_DISABLE_UNINSTALLED', 'PKG_CONFIG_LIBDIR', 'PKG_CONFIG_PATH', 'PKG_CONFIG_SYSROOT_DIR', 'PN', 'PR', 'PSEUDO_DISABLED', 'PV', 'QPEDIR', 'QTDIR', 'RANLIB', 'S', 'SELECTED_OPTIMIZATION', 'SITECONFIG_SYSROOTCACHE', 'SRCPV', 'STAGING_DATADIR', 'STAGING_DIR', 'STAGING_DIR_NATIVE', 'STAGING_INCDIR_NATIVE', 'STAGING_LIBDIR_NATIVE', 'STRIP', 'TARGET_ARCH', 'TARGET_CFLAGS', 'TARGET_CPPFLAGS', 'TARGET_CXXFLAGS', 'TARGET_LDFLAGS', 'TARGET_LINK_HASH_STYLE', 'TARGET_OS', 'TARGET_SYS', 'TARGET_VENDOR', 'TOOLCHAIN_OPTIONS', 'TUNE_ARCH', 'TUNE_PKGARCH', 'base_bindir', 'base_libdir', 'base_prefix', 'base_sbindir', 'baselib', 'bindir', 'datadir', 'docdir', 'exec_prefix', 'includedir', 'includedir_native', 'infodir', 'libdir', 'libdir_native', 'libexecdir', 'localstatedir', 'lt_cv_sys_lib_dlsearch_path_spec', 'mandir', 'oldincludedir', 'palmqtdir', 'palmtopdir', 'prefix', 'prefix_native', 'sbindir', 'servicedir', 'sharedstatedir', 'siteinfo_data', 'siteinfo_get_files', 'sysconfdir']
to:
['ABIEXTENSION', 'AR', 'ARMPKGARCH', 'ARMPKGSFX_DSP', 'ARMPKGSFX_EABI', 'ARMPKGSFX_ENDIAN', 'ARMPKGSFX_FPU', 'ARMPKGSFX_THUMB', 'ARM_INSTRUCTION_SET', 'ARM_THUMB_M_OPT', 'AS', 'ASNEEDED', 'BRANCH', 'BUILD_AR', 'BUILD_ARCH', 'BUILD_AS', 'BUILD_AS_ARCH', 'BUILD_CC', 'BUILD_CCLD', 'BUILD_CC_ARCH', 'BUILD_CFLAGS', 'BUILD_CPP', 'BUILD_CPPFLAGS', 'BUILD_CXX', 'BUILD_CXXFLAGS', 'BUILD_F77', 'BUILD_LD', 'BUILD_LDFLAGS', 'BUILD_LD_ARCH', 'BUILD_NM', 'BUILD_OPTIMIZATION', 'BUILD_OS', 'BUILD_PREFIX', 'BUILD_RANLIB', 'BUILD_STRIP', 'BUILD_SYS', 'BUILD_VENDOR', 'CC', 'CCACHE', 'CCACHE_DIR', 'CCLD', 'CFLAGS', 'CONFIG_SITE', 'CPP', 'CPPFLAGS', 'CXX', 'CXXFLAGS', 'DEBUG_BUILD', 'DEBUG_FLAGS', 'DEBUG_OPTIMIZATION', 'F77', 'FULL_OPTIMIZATION', 'HOST_ARCH', 'HOST_AS_ARCH', 'HOST_CC_ARCH', 'HOST_LD_ARCH', 'HOST_OS', 'HOST_PREFIX', 'HOST_VENDOR', 'LC_ALL', 'LD', 'LDFLAGS', 'LIBCEXTENSION', 'LINKER_HASH_STYLE', 'MAKE', 'MULTIMACH_HOST_SYS', 'NM', 'OBJCOPY', 'OBJDUMP', 'OPIEDIR', 'PACKAGE_ARCH', 'PATCH_GET', 'PKG_CONFIG_DIR', 'PKG_CONFIG_DISABLE_UNINSTALLED', 'PKG_CONFIG_LIBDIR', 'PKG_CONFIG_PATH', 'PKG_CONFIG_SYSROOT_DIR', 'PN', 'PR', 'PSEUDO_DISABLED', 'PV', 'QPEDIR', 'QTDIR', 'RANLIB', 'S', 'SELECTED_OPTIMIZATION', 'SITECONFIG_SYSROOTCACHE', 'SRCPV', 'STAGING_DATADIR', 'STAGING_DIR', 'STAGING_DIR_NATIVE', 'STAGING_INCDIR_NATIVE', 'STAGING_LIBDIR_NATIVE', 'STRIP', 'TARGET_ARCH', 'TARGET_AS_ARCH', 'TARGET_CC_ARCH', 'TARGET_CFLAGS', 'TARGET_CPPFLAGS', 'TARGET_CXXFLAGS', 'TARGET_LDFLAGS', 'TARGET_LD_ARCH', 'TARGET_LINK_HASH_STYLE', 'TARGET_OS', 'TARGET_PREFIX', 'TARGET_SYS', 'TARGET_VENDOR', 'TOOLCHAIN_OPTIONS', 'TUNE_ARCH', 'TUNE_ASARGS', 'TUNE_CCARGS', 'TUNE_LDARGS', 'TUNE_PKGARCH', 'base_bindir', 'base_libdir', 'base_prefix', 'base_sbindir', 'baselib', 'bindir', 'datadir', 'docdir', 'exec_prefix', 'includedir', 'includedir_native', 'infodir', 'libdir', 'libdir_native', 'libexecdir', 'localstatedir', 'lt_cv_sys_lib_dlsearch_path_spec', 'mandir', 'oldincludedir', 'palmqtdir', 'palmtopdir', 'prefix', 'prefix_native', 'sbindir', 'servicedir', 'sharedstatedir', 'siteinfo_data', 'siteinfo_get_files', 'sysconfdir']
basehash changed from e593cd201ee425228fc7eea0dc86c848 to 00eacaec12ed71089c4a54c532ce9996
List of dependencies for variable CXXFLAGS changed from set(['BUILD_CFLAGS']) to set([])
changed items: set(['BUILD_CFLAGS'])
List of dependencies for variable prefix changed from set(['STAGING_DIR_NATIVE', 'prefix_native']) to set([])
changed items: set(['STAGING_DIR_NATIVE', 'prefix_native'])
List of dependencies for variable HOST_OS changed from set(['BUILD_OS']) to set(['TARGET_OS'])
changed items: set(['TARGET_OS', 'BUILD_OS'])
List of dependencies for variable HOST_VENDOR changed from set(['BUILD_VENDOR']) to set(['TARGET_VENDOR'])
changed items: set(['TARGET_VENDOR', 'BUILD_VENDOR'])
List of dependencies for variable base_bindir changed from set(['bindir']) to set(['base_prefix'])
changed items: set(['base_prefix', 'bindir'])
List of dependencies for variable HOST_CC_ARCH changed from set(['BUILD_CC_ARCH']) to set(['TARGET_CC_ARCH'])
changed items: set(['BUILD_CC_ARCH', 'TARGET_CC_ARCH'])
List of dependencies for variable PKG_CONFIG_DIR changed from set(['STAGING_DIR', 'TUNE_PKGARCH', 'libdir', 'TARGET_VENDOR', 'TARGET_OS']) to set(['STAGING_DIR_HOST', 'libdir'])
changed items: set(['STAGING_DIR_HOST', 'STAGING_DIR', 'TARGET_VENDOR', 'TUNE_PKGARCH', 'TARGET_OS'])
List of dependencies for variable libdir changed from set(['exec_prefix', 'CROSS_TARGET_SYS_DIR']) to set(['exec_prefix', 'baselib'])
changed items: set(['CROSS_TARGET_SYS_DIR', 'baselib'])
List of dependencies for variable LDFLAGS changed from set(['BUILD_LDFLAGS']) to set(['TARGET_LDFLAGS'])
changed items: set(['TARGET_LDFLAGS', 'BUILD_LDFLAGS'])
List of dependencies for variable HOST_LD_ARCH changed from set(['BUILD_LD_ARCH']) to set(['TARGET_LD_ARCH'])
changed items: set(['BUILD_LD_ARCH', 'TARGET_LD_ARCH'])
List of dependencies for variable base_prefix changed from set(['STAGING_DIR_NATIVE']) to set([])
changed items: set(['STAGING_DIR_NATIVE'])
List of dependencies for variable bindir changed from set(['exec_prefix', 'CROSS_TARGET_SYS_DIR']) to set(['exec_prefix'])
changed items: set(['CROSS_TARGET_SYS_DIR'])
List of dependencies for variable HOST_ARCH changed from set(['BUILD_ARCH']) to set(['TARGET_ARCH'])
changed items: set(['BUILD_ARCH', 'TARGET_ARCH'])
List of dependencies for variable PKG_CONFIG_SYSROOT_DIR changed from set(['STAGING_DIR', 'TUNE_PKGARCH', 'TARGET_VENDOR', 'TARGET_OS']) to set(['STAGING_DIR_HOST'])
changed items: set(['STAGING_DIR', 'TUNE_PKGARCH', 'STAGING_DIR_HOST', 'TARGET_VENDOR', 'TARGET_OS'])
List of dependencies for variable base_sbindir changed from set(['bindir']) to set(['base_prefix'])
changed items: set(['base_prefix', 'bindir'])
List of dependencies for variable HOST_PREFIX changed from set(['BUILD_PREFIX']) to set(['TARGET_PREFIX'])
changed items: set(['BUILD_PREFIX', 'TARGET_PREFIX'])
List of dependencies for variable TOOLCHAIN_OPTIONS changed from set([]) to set(['STAGING_DIR_TARGET'])
changed items: set(['STAGING_DIR_TARGET'])
List of dependencies for variable sbindir changed from set(['bindir']) to set(['exec_prefix'])
changed items: set(['exec_prefix', 'bindir'])
List of dependencies for variable CFLAGS changed from set(['BUILD_CFLAGS']) to set(['TARGET_CFLAGS'])
changed items: set(['TARGET_CFLAGS', 'BUILD_CFLAGS'])
List of dependencies for variable exec_prefix changed from set(['STAGING_DIR_NATIVE', 'prefix_native']) to set([])
changed items: set(['STAGING_DIR_NATIVE', 'prefix_native'])
List of dependencies for variable HOST_AS_ARCH changed from set(['BUILD_AS_ARCH']) to set(['TARGET_AS_ARCH'])
changed items: set(['TARGET_AS_ARCH', 'BUILD_AS_ARCH'])
List of dependencies for variable libexecdir changed from set(['exec_prefix', 'CROSS_TARGET_SYS_DIR']) to set(['exec_prefix'])
changed items: set(['CROSS_TARGET_SYS_DIR'])
Dependency on variable CROSS_TARGET_SYS_DIR was added
Dependency on Variable TUNE_CCARGS was removed
Dependency on Variable ARM_INSTRUCTION_SET was removed
Dependency on Variable TARGET_AS_ARCH was removed
Dependency on Variable ARM_THUMB_M_OPT was removed
Dependency on Variable TARGET_PREFIX was removed
Dependency on Variable TUNE_LDARGS was removed
Dependency on Variable TUNE_ASARGS was removed
Dependency on Variable TARGET_LD_ARCH was removed
Dependency on Variable TARGET_CC_ARCH was removed
Variable CXXFLAGS value changed from ${BUILD_CFLAGS} to -O2 -pipe -g -feliminate-unused-debug-types -fpermissive
Variable prefix value changed from ${STAGING_DIR_NATIVE}${prefix_native} to /usr
Variable HOST_OS value changed from ${BUILD_OS} to ${TARGET_OS}
Variable HOST_VENDOR value changed from ${BUILD_VENDOR} to ${TARGET_VENDOR}
Variable base_bindir value changed from ${bindir} to ${base_prefix}/bin
Variable HOST_CC_ARCH value changed from ${BUILD_CC_ARCH} to ${TARGET_CC_ARCH}
Variable PKG_CONFIG_DIR value changed from ${STAGING_DIR}/${TUNE_PKGARCH}${TARGET_VENDOR}-${TARGET_OS}${libdir}/pkgconfig to ${STAGING_DIR_HOST}/${libdir}/pkgconfig
Variable libdir value changed from ${exec_prefix}/lib/${CROSS_TARGET_SYS_DIR} to ${exec_prefix}/${baselib}
Variable LDFLAGS value changed from ${BUILD_LDFLAGS} to ${TARGET_LDFLAGS}
Variable HOST_LD_ARCH value changed from ${BUILD_LD_ARCH} to ${TARGET_LD_ARCH}
Variable base_prefix value changed from ${STAGING_DIR_NATIVE} to
Variable bindir value changed from ${exec_prefix}/bin/${CROSS_TARGET_SYS_DIR} to ${exec_prefix}/bin
Variable HOST_ARCH value changed from ${BUILD_ARCH} to ${TARGET_ARCH}
Variable PKG_CONFIG_SYSROOT_DIR value changed from ${STAGING_DIR}/${TUNE_PKGARCH}${TARGET_VENDOR}-${TARGET_OS} to ${STAGING_DIR_HOST}
Variable base_sbindir value changed from ${bindir} to ${base_prefix}/sbin
Variable HOST_PREFIX value changed from ${BUILD_PREFIX} to ${TARGET_PREFIX}
Variable TOOLCHAIN_OPTIONS value changed from  to  --sysroot=${STAGING_DIR_TARGET}
Variable sbindir value changed from ${bindir} to ${exec_prefix}/sbin
Variable CFLAGS value changed from ${BUILD_CFLAGS} to ${TARGET_CFLAGS}
Variable exec_prefix value changed from ${STAGING_DIR_NATIVE}${prefix_native} to /usr
Variable HOST_AS_ARCH value changed from ${BUILD_AS_ARCH} to ${TARGET_AS_ARCH}
Variable libexecdir value changed from ${exec_prefix}/libexec/${CROSS_TARGET_SYS_DIR} to ${exec_prefix}/libexec
Dependency on task gcc-cross-initial_4.6.bb.do_unpack was added
Dependency on task gcc-runtime_4.6.bb.do_unpack was removed

Best Regards,
Lianhao

Yes, this is exactly where this difference is coming from.

The problem is that do_headerfix has a lot of variables exported into its environment, even if it doesn't need/use them. I'm wondering if we should override bitbake in this particular case using the vardeps flag with something like:

do_headerfix[vardeps] = "PATH"

which might solve this problem cleanly and remove the need for my original patch too...

Cheers,

Richard

this patch can be dropped we dont build fortran at present if we need to then this should be fixed via configure variable.

-Khem