Major feature changes for the next release - merging very soon
As has been mentioned in other emails we're working on some major enhancements for the next release. Several components of these are now ready for merging to the master branch. These include:
* Rework of the compiler bootstrap process
If you've ever tried to run "bitbake eglibc -c clean;bitbake eglibc" or any of the gcc components you'll have run into issues with the compiler bootstrap process overwriting files. This also causes problems for sstate as it doesn't get on well with two sstate archives providing the same file.
The changes we have queued split the intermediate components into a separate sysroot so no files are overwritten and the whole process is less fragile and more robust. Each gcc-cross step (initial, intermediate and final) also stage binaries in separate locations.
* Changes to sysroot structure
We now support creating a sysroot per machine target rather than the original multimachine approach we have used. This means packages with an "all" architecture can be safely installed into the sysroot and used correctly fixing bugs in that area and it also allows machines like emenlow which have separate graphics components to build and work correctly yet be able to share builds with other machines like atom-pc.
If you change machine and the machine you change to shares a core architecture with a previous build, the components from that previous build will be used to construct a new sysroot using the sstate prebuilt packages.
The branch I'm planning to merge very soon with these features is:
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=rpurdie/tt-bootstrap
There are also some changes in there to fix pseudo issues people have been seeing and there are some fetcher changes in there moving us towards more reliability in the fetcher.
Due to the nature of the changes, people are going to have to wipe tmpdir and rebuild after these changes and anyone trying to use an existing build will see an error message. We try hard to avoid requiring this but this is a side effect of making significant structural improvements like the above. Just to be 100% safe I changed the sstate version number too, invalidating previous sstate packages due to the structure changes.
Since this is a new development there may be a few bumps as we merge things, I think these developments are exciting and fix long standing problems so please bare with us as we work though any issues!
Cheers,
Richard
Op 25 jan 2011, om 14:57 heeft Richard Purdie het volgende geschreven:
Quoted 26 lines
As has been mentioned in other emails we're working on some major enhancements for the next release. Several components of these are now ready for merging to the master branch. These include:
* Rework of the compiler bootstrap process
If you've ever tried to run "bitbake eglibc -c clean;bitbake eglibc" or any of the gcc components you'll have run into issues with the compiler bootstrap process overwriting files. This also causes problems for sstate as it doesn't get on well with two sstate archives providing the same file.
The changes we have queued split the intermediate components into a separate sysroot so no files are overwritten and the whole process is less fragile and more robust. Each gcc-cross step (initial, intermediate and final) also stage binaries in separate locations.
* Changes to sysroot structure
We now support creating a sysroot per machine target rather than the original multimachine approach we have used. This means packages with an "all" architecture can be safely installed into the sysroot and used correctly fixing bugs in that area and it also allows machines like emenlow which have separate graphics components to build and work correctly yet be able to share builds with other machines like atom-pc.
Quoted 8 lines
If you change machine and the machine you change to shares a core architecture with a previous build, the components from that previous build will be used to construct a new sysroot using the sstate prebuilt packages.
The branch I'm planning to merge very soon with these features is:
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=rpurdie/tt-bootstrap
My build of console-image for angstrom using that branch is not at:
NOTE: package gcc-cross-4.5-r31.2+svnr168622: task do_package: Succeeded
So far, so good :)
regards,
Koen
Op 25 jan 2011, om 14:57 heeft Richard Purdie het volgende geschreven:
Quoted 26 lines
As has been mentioned in other emails we're working on some major enhancements for the next release. Several components of these are now ready for merging to the master branch. These include:
* Rework of the compiler bootstrap process
If you've ever tried to run "bitbake eglibc -c clean;bitbake eglibc" or any of the gcc components you'll have run into issues with the compiler bootstrap process overwriting files. This also causes problems for sstate as it doesn't get on well with two sstate archives providing the same file.
The changes we have queued split the intermediate components into a separate sysroot so no files are overwritten and the whole process is less fragile and more robust. Each gcc-cross step (initial, intermediate and final) also stage binaries in separate locations.
* Changes to sysroot structure
We now support creating a sysroot per machine target rather than the original multimachine approach we have used. This means packages with an "all" architecture can be safely installed into the sysroot and used correctly fixing bugs in that area and it also allows machines like emenlow which have separate graphics components to build and work correctly yet be able to share builds with other machines like atom-pc.
Quoted 8 lines
If you change machine and the machine you change to shares a core architecture with a previous build, the components from that previous build will be used to construct a new sysroot using the sstate prebuilt packages.
The branch I'm planning to merge very soon with these features is:
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=rpurdie/tt-bootstrap
For a single machine:
NOTE: package console-image-1.0-r0: task do_rm_work_all: Succeeded
Now building it incrementally for a different machine
Op 25 jan 2011, om 19:34 heeft Koen Kooi het volgende geschreven:
Quoted 36 lines
Op 25 jan 2011, om 14:57 heeft Richard Purdie het volgende geschreven:
Quoted 33 lines
As has been mentioned in other emails we're working on some major enhancements for the next release. Several components of these are now ready for merging to the master branch. These include:
* Rework of the compiler bootstrap process
If you've ever tried to run "bitbake eglibc -c clean;bitbake eglibc" or any of the gcc components you'll have run into issues with the compiler bootstrap process overwriting files. This also causes problems for sstate as it doesn't get on well with two sstate archives providing the same file.
The changes we have queued split the intermediate components into a separate sysroot so no files are overwritten and the whole process is less fragile and more robust. Each gcc-cross step (initial, intermediate and final) also stage binaries in separate locations.
* Changes to sysroot structure
We now support creating a sysroot per machine target rather than the original multimachine approach we have used. This means packages with an "all" architecture can be safely installed into the sysroot and used correctly fixing bugs in that area and it also allows machines like emenlow which have separate graphics components to build and work correctly yet be able to share builds with other machines like atom-pc.
If you change machine and the machine you change to shares a core architecture with a previous build, the components from that previous build will be used to construct a new sysroot using the sstate prebuilt packages.
The branch I'm planning to merge very soon with these features is:
Quoted 8 lines
For a single machine:
NOTE: package console-image-1.0-r0: task
do_rm_work_all: SucceededNow building it incrementally for a different machine
Switchting from usrp-e1xx (armv7a) to beagleboard (also armv7a) and building console-image again:
NOTE: package perl-5.12.2-r0: task do_package: Started
ERROR: Function 'perl_package_preprocess' failed (see /OE/tentacle/build/tmp-angstrom_2010_x/work/armv7a-angstrom-linux-gnueabi/perl-5.12.2-r0/temp/log.do_package.28537 for further information)
ERROR: Logfile of failure stored in: /OE/tentacle/build/tmp-angstrom_2010_x/work/armv7a-angstrom-linux-gnueabi/perl-5.12.2-r0/temp/log.do_package.28537
Log data follows:
| cp: cannot stat `/OE/tentacle/build/tmp-angstrom_2010_x/work/armv7a-angstrom-linux-gnueabi/perl-5.12.2-r0/image/*': No such file or directory | sed: can't read /OE/tentacle/build/tmp-angstrom_2010_x/work/armv7a-angstrom-linux-gnueabi/perl-5.12.2-r0/package/usr/bin/h2xs: No such file or directory | sed: can't read /OE/tentacle/build/tmp-angstrom_2010_x/work/armv7a-angstrom-linux-gnueabi/perl-5.12.2-r0/package/usr/bin/h2ph: No such file or directory | sed: can't read /OE/tentacle/build/tmp-angstrom_2010_x/work/armv7a-angstrom-linux-gnueabi/perl-5.12.2-r0/package/usr/lib/perl/5.12.2/pod/*.pod: No such file or directory | sed: can't read /OE/tentacle/build/tmp-angstrom_2010_x/work/armv7a-angstrom-linux-gnueabi/perl-5.12.2-r0/package/usr/lib/perl/5.12.2/cacheout.pl: No such file or directory | sed: can't read /OE/tentacle/build/tmp-angstrom_2010_x/work/armv7a-angstrom-linux-gnueabi/perl-5.12.2-r0/package/usr/lib/perl/5.12.2/FileCache.pm: No such file or directory | sed: can't read /OE/tentacle/build/tmp-angstrom_2010_x/work/armv7a-angstrom-linux-gnueabi/perl-5.12.2-r0/package/usr/lib/perl/config.sh: No such file or directory | sed: can't read /OE/tentacle/build/tmp-angstrom_2010_x/work/armv7a-angstrom-linux-gnueabi/perl-5.12.2-r0/package/usr/lib/perl/5.12.2/Config.pm: No such file or directory | sed: can't read /OE/tentacle/build/tmp-angstrom_2010_x/work/armv7a-angstrom-linux-gnueabi/perl-5.12.2-r0/package/usr/lib/perl/5.12.2/Config_heavy.pl: No such file or directory | sed: can't read /OE/tentacle/build/tmp-angstrom_2010_x/work/armv7a-angstrom-linux-gnueabi/perl-5.12.2-r0/package/usr/lib/perl/5.12.2/CORE/perl.h: No such file or directory | sed: can't read /OE/tentacle/build/tmp-angstrom_2010_x/work/armv7a-angstrom-linux-gnueabi/perl-5.12.2-r0/package/usr/lib/perl/5.12.2/CORE/pp.h: No such file or directory | Function 'perl_package_preprocess' failed (see /OE/tentacle/build/tmp-angstrom_2010_x/work/armv7a-angstrom-linux-gnueabi/perl-5.12.2-r0/temp/log.do_package.28537 for further information) | ERROR: Function 'perl_package_preprocess' failed (see /OE/tentacle/build/tmp-angstrom_2010_x/work/armv7a-angstrom-linux-gnueabi/perl-5.12.2-r0/temp/log.do_package.28537 for further information)
Op 25 jan 2011, om 20:18 heeft Koen Kooi het volgende geschreven:
Quoted 54 lines
Op 25 jan 2011, om 19:34 heeft Koen Kooi het volgende geschreven:
Quoted 36 lines
Op 25 jan 2011, om 14:57 heeft Richard Purdie het volgende geschreven:
Quoted 33 lines
As has been mentioned in other emails we're working on some major enhancements for the next release. Several components of these are now ready for merging to the master branch. These include:
* Rework of the compiler bootstrap process
If you've ever tried to run "bitbake eglibc -c clean;bitbake eglibc" or any of the gcc components you'll have run into issues with the compiler bootstrap process overwriting files. This also causes problems for sstate as it doesn't get on well with two sstate archives providing the same file.
The changes we have queued split the intermediate components into a separate sysroot so no files are overwritten and the whole process is less fragile and more robust. Each gcc-cross step (initial, intermediate and final) also stage binaries in separate locations.
* Changes to sysroot structure
We now support creating a sysroot per machine target rather than the original multimachine approach we have used. This means packages with an "all" architecture can be safely installed into the sysroot and used correctly fixing bugs in that area and it also allows machines like emenlow which have separate graphics components to build and work correctly yet be able to share builds with other machines like atom-pc.
If you change machine and the machine you change to shares a core architecture with a previous build, the components from that previous build will be used to construct a new sysroot using the sstate prebuilt packages.
The branch I'm planning to merge very soon with these features is:
Quoted 8 lines
For a single machine:
NOTE: package console-image-1.0-r0: task
do_rm_work_all: SucceededNow building it incrementally for a different machine
Switchting from usrp-e1xx (armv7a) to beagleboard (also armv7a) and building console-image again:
NOTE: package perl-5.12.2-r0: task
do_package: Started
ERROR: Function 'perl_package_preprocess' failed (see /OE/tentacle/build/tmp-angstrom_2010_x/work/armv7a-angstrom-linux-gnueabi/perl-5.12.2-r0/temp/log.do_package.28537 for further information)
ERROR: Logfile of failure stored in: /OE/tentacle/build/tmp-angstrom_2010_x/work/armv7a-angstrom-linux-gnueabi/perl-5.12.2-r0/temp/log.do_package.28537
Log data follows:
Quoted 13 lines
| cp: cannot stat `/OE/tentacle/build/tmp-angstrom_2010_x/work/armv7a-angstrom-linux-gnueabi/perl-5.12.2-r0/image/*': No such file or directory | sed: can't read /OE/tentacle/build/tmp-angstrom_2010_x/work/armv7a-angstrom-linux-gnueabi/perl-5.12.2-r0/package/usr/bin/h2xs: No such file or directory | sed: can't read /OE/tentacle/build/tmp-angstrom_2010_x/work/armv7a-angstrom-linux-gnueabi/perl-5.12.2-r0/package/usr/bin/h2ph: No such file or directory | sed: can't read /OE/tentacle/build/tmp-angstrom_2010_x/work/armv7a-angstrom-linux-gnueabi/perl-5.12.2-r0/package/usr/lib/perl/5.12.2/pod/*.pod: No such file or directory | sed: can't read /OE/tentacle/build/tmp-angstrom_2010_x/work/armv7a-angstrom-linux-gnueabi/perl-5.12.2-r0/package/usr/lib/perl/5.12.2/cacheout.pl: No such file or directory | sed: can't read /OE/tentacle/build/tmp-angstrom_2010_x/work/armv7a-angstrom-linux-gnueabi/perl-5.12.2-r0/package/usr/lib/perl/5.12.2/FileCache.pm: No such file or directory | sed: can't read /OE/tentacle/build/tmp-angstrom_2010_x/work/armv7a-angstrom-linux-gnueabi/perl-5.12.2-r0/package/usr/lib/perl/config.sh: No such file or directory | sed: can't read /OE/tentacle/build/tmp-angstrom_2010_x/work/armv7a-angstrom-linux-gnueabi/perl-5.12.2-r0/package/usr/lib/perl/5.12.2/Config.pm: No such file or directory | sed: can't read /OE/tentacle/build/tmp-angstrom_2010_x/work/armv7a-angstrom-linux-gnueabi/perl-5.12.2-r0/package/usr/lib/perl/5.12.2/Config_heavy.pl: No such file or directory | sed: can't read /OE/tentacle/build/tmp-angstrom_2010_x/work/armv7a-angstrom-linux-gnueabi/perl-5.12.2-r0/package/usr/lib/perl/5.12.2/CORE/perl.h: No such file or directory | sed: can't read /OE/tentacle/build/tmp-angstrom_2010_x/work/armv7a-angstrom-linux-gnueabi/perl-5.12.2-r0/package/usr/lib/perl/5.12.2/CORE/pp.h: No such file or directory | Function 'perl_package_preprocess' failed (see /OE/tentacle/build/tmp-angstrom_2010_x/work/armv7a-angstrom-linux-gnueabi/perl-5.12.2-r0/temp/log.do_package.28537 for further information) | ERROR: Function 'perl_package_preprocess' failed (see /OE/tentacle/build/tmp-angstrom_2010_x/work/armv7a-angstrom-linux-gnueabi/perl-5.12.2-r0/temp/log.do_package.28537 for further information)
And another one:
NOTE: package pango-1.28.3-r1.0: task do_package: Started
ERROR: Error executing a python function in /OE/tentacle/sources/layers/meta-openembedded/recipes-graphics/pango/pango_1.28.3.bb:
OSError: [Errno 2] No such file or directory: '/OE/tentacle/build/tmp-angstrom_2010_x/work/armv7a-angstrom-linux-gnueabi/pango-1.28.3-r1.0/package/usr/lib/pango/1.6.0/modules'
ERROR: The stack trace of python calls that resulted in this exception/failure was:
ERROR: File "populate_packages", line 203, in <module>
ERROR:
ERROR: File "populate_packages", line 7, in populate_packages
ERROR:
ERROR: File "package.bbclass", line 16, in do_split_packages
ERROR:
ERROR: The code that was being executed was:
ERROR: 0199: bb.data.setVar('pkg_prerm_%s' % pkg, prerm, d)
ERROR: 0200:
ERROR: 0201:
ERROR: 0202:
ERROR: *** 0203:populate_packages(d)
ERROR: 0204:
ERROR: (file: 'populate_packages', lineno: 203, function: <module>)
ERROR: 0003: prologue = bb.data.getVar("postinst_prologue", d, 1)
ERROR: 0004:
ERROR: 0005: modules_root = bb.data.expand('/usr/lib/pango/1.6.0/modules', d)
ERROR: 0006:
ERROR: *** 0007: do_split_packages(d, modules_root, '^pango-(.*)\.so$', 'pango-module-%s', 'Pango module %s', prologue + 'pango-querymodules > /etc/pango/pango.modules')
ERROR: 0008: import glob, stat, errno, re
ERROR: 0009:
ERROR: 0010: workdir = bb.data.getVar('WORKDIR', d, True)
ERROR: 0011: outdir = bb.data.getVar('DEPLOY_DIR', d, True)
ERROR: (file: 'populate_packages', lineno: 7, function: populate_packages)
ERROR: Function 'populate_packages' failed
ERROR: Logfile of failure stored in: /OE/tentacle/build/tmp-angstrom_2010_x/work/armv7a-angstrom-linux-gnueabi/pango-1.28.3-r1.0/temp/log.do_package.27054It looks like do_install needs to get rerun on those recipes.
Quoted 55 lines · Koen Kooi
Op 25 jan 2011, om 20:18 heeft Koen Kooi het volgende geschreven:
Quoted 6 lines
Switchting from usrp-e1xx (armv7a) to beagleboard (also armv7a) and building console-image again:
NOTE: package perl-5.12.2-r0: task
do_package: Started
ERROR: Function 'perl_package_preprocess' failed (see /OE/tentacle/build/tmp-angstrom_2010_x/work/armv7a-angstrom-linux-gnueabi/perl-5.12.2-r0/temp/log.do_package.28537 for further information)
ERROR: Logfile of failure stored in: /OE/tentacle/build/tmp-angstrom_2010_x/work/armv7a-angstrom-linux-gnueabi/perl-5.12.2-r0/temp/log.do_package.28537
Log data follows:Quoted 13 lines
| cp: cannot stat `/OE/tentacle/build/tmp-angstrom_2010_x/work/armv7a-angstrom-linux-gnueabi/perl-5.12.2-r0/image/*': No such file or directory | sed: can't read /OE/tentacle/build/tmp-angstrom_2010_x/work/armv7a-angstrom-linux-gnueabi/perl-5.12.2-r0/package/usr/bin/h2xs: No such file or directory | sed: can't read /OE/tentacle/build/tmp-angstrom_2010_x/work/armv7a-angstrom-linux-gnueabi/perl-5.12.2-r0/package/usr/bin/h2ph: No such file or directory | sed: can't read /OE/tentacle/build/tmp-angstrom_2010_x/work/armv7a-angstrom-linux-gnueabi/perl-5.12.2-r0/package/usr/lib/perl/5.12.2/pod/*.pod: No such file or directory | sed: can't read /OE/tentacle/build/tmp-angstrom_2010_x/work/armv7a-angstrom-linux-gnueabi/perl-5.12.2-r0/package/usr/lib/perl/5.12.2/cacheout.pl: No such file or directory | sed: can't read /OE/tentacle/build/tmp-angstrom_2010_x/work/armv7a-angstrom-linux-gnueabi/perl-5.12.2-r0/package/usr/lib/perl/5.12.2/FileCache.pm: No such file or directory | sed: can't read /OE/tentacle/build/tmp-angstrom_2010_x/work/armv7a-angstrom-linux-gnueabi/perl-5.12.2-r0/package/usr/lib/perl/config.sh: No such file or directory | sed: can't read /OE/tentacle/build/tmp-angstrom_2010_x/work/armv7a-angstrom-linux-gnueabi/perl-5.12.2-r0/package/usr/lib/perl/5.12.2/Config.pm: No such file or directory | sed: can't read /OE/tentacle/build/tmp-angstrom_2010_x/work/armv7a-angstrom-linux-gnueabi/perl-5.12.2-r0/package/usr/lib/perl/5.12.2/Config_heavy.pl: No such file or directory | sed: can't read /OE/tentacle/build/tmp-angstrom_2010_x/work/armv7a-angstrom-linux-gnueabi/perl-5.12.2-r0/package/usr/lib/perl/5.12.2/CORE/perl.h: No such file or directory | sed: can't read /OE/tentacle/build/tmp-angstrom_2010_x/work/armv7a-angstrom-linux-gnueabi/perl-5.12.2-r0/package/usr/lib/perl/5.12.2/CORE/pp.h: No such file or directory | Function 'perl_package_preprocess' failed (see /OE/tentacle/build/tmp-angstrom_2010_x/work/armv7a-angstrom-linux-gnueabi/perl-5.12.2-r0/temp/log.do_package.28537 for further information) | ERROR: Function 'perl_package_preprocess' failed (see /OE/tentacle/build/tmp-angstrom_2010_x/work/armv7a-angstrom-linux-gnueabi/perl-5.12.2-r0/temp/log.do_package.28537 for further information)And another one:
NOTE: package pango-1.28.3-r1.0: task
do_package: Started
ERROR: Error executing a python function in /OE/tentacle/sources/layers/meta-openembedded/recipes-graphics/pango/pango_1.28.3.bb:
OSError: [Errno 2] No such file or directory: '/OE/tentacle/build/tmp-angstrom_2010_x/work/armv7a-angstrom-linux-gnueabi/pango-1.28.3-r1.0/package/usr/lib/pango/1.6.0/modules'ERROR: The stack trace of python calls that resulted in this exception/failure was:
ERROR: File "populate_packages", line 203, in <module>
ERROR:
ERROR: File "populate_packages", line 7, inpopulate_packages
ERROR:
ERROR: File "package.bbclass", line 16, indo_split_packages
ERROR:ERROR: The code that was being executed was: ERROR: 0199: bb.data.setVar('pkg_prerm_%s' % pkg, prerm, d) ERROR: 0200: ERROR: 0201: ERROR: 0202: ERROR: *** 0203:populate_packages(d) ERROR: 0204: ERROR: (file: 'populate_packages', lineno: 203, function: <module>) ERROR: 0003: prologue = bb.data.getVar("postinst_prologue", d, 1) ERROR: 0004: ERROR: 0005: modules_root = bb.data.expand('/usr/lib/pango/1.6.0/modules', d) ERROR: 0006: ERROR: *** 0007: do_split_packages(d, modules_root, '^pango-(.*)\.so$', 'pango-module-%s', 'Pango module %s', prologue + 'pango-querymodules > /etc/pango/pango.modules') ERROR: 0008: import glob, stat, errno, re ERROR: 0009: ERROR: 0010: workdir = bb.data.getVar('WORKDIR', d, True) ERROR: 0011: outdir = bb.data.getVar('DEPLOY_DIR', d, True) ERROR: (file: 'populate_packages', lineno: 7, function: populate_packages) ERROR: Function 'populate_packages' failed ERROR: Logfile of failure stored in: /OE/tentacle/build/tmp-angstrom_2010_x/work/armv7a-angstrom-linux-gnueabi/pango-1.28.3-r1.0/temp/log.do_package.27054
Quoted 1 line
It looks like
do_installneeds to get rerun on those recipes.
I think these are due to the fact Koen is using rm_work (confirmed by Koen over irc). I agree we need to work out a way to handle this, it should help anyone trying to find a fix though.
It should be easy to reproduce on Poky master switching between emenlow and atom-pc.
Cheers,
Richard