Merging problems
I want to give people a headsup that we're having problems merging changes at the moment. We've been doing our best but the number of things building up which are causing issues is overwheling our ability to fix and stablise the build. It wasn't helped that I took a long weekend's vacation last weekend. There are changes being made or tested to the autobuilder which also isn't helping.
The kernel series has several issues:
* a random failure in do_kernel_configme [i]
* bzImage not being built for some machines [ii]
* non linux-yocto kernel failures [iii]
* the headers upgrade breaks socat (fix pending, thanks Bruce)
There was a change merged to bitbake which broken cups on lsb and powertop due to unset variable problems. I have two pending fixes for those.
For the last test build, we have 195 failures to dig through:
and we've not managed to dive into many of them. I don't have fixes for the three kernel issues above.
So I'd ask that people be patient and if you have some time to help with any of these problems, I'd welcome it. Most people are off on vacation now which is going to leave me with a challenge.
Cheers,
Richard
[i] https://autobuilder.yoctoproject.org/main/builders/nightly-ppc-lsb/builds/129/steps/BuildImages/logs/stdio
[ii] https://autobuilder.yoctoproject.org/main/builders/minnow/builds/129/steps/BuildImages/logs/stdio
[iii] https://autobuilder.yoctoproject.org/main/builders/nightly-fsl-arm/builds/128/steps/BuildImages/logs/stdio
(5 unknown build failures in this one!)
https://autobuilder.yoctoproject.org/main/builders/nightly-fsl-ppc/builds/129/steps/BuildImages/logs/stdio
(only 3 here)
Quoted 10 lines · Richard Purdie
I want to give people a headsup that we're having problems merging changes at the moment. We've been doing our best but the number of things building up which are causing issues is overwheling our ability to fix and stablise the build. It wasn't helped that I took a long weekend's vacation last weekend. There are changes being made or tested to the autobuilder which also isn't helping.
The kernel series has several issues:
* a random failure in
do_kernel_configme[i]
I think I have a handle on this. If you look at the autobuilder failure it says:
| [INFO] Configuring target/machine combo: "standard/qemuppc" | [INFO] collecting configs in patches/meta-series
and what concerns me is "patches/meta-series". I my local builds it says .meta/meta-series. Poking around kern-tools I see:
"""
# determine the meta directory name. The meta directory is at the top level
# of the repository, and is untracked.
meta_dir_options=`git ls-files -o --directory`
for m in $meta_dir_options; do
if [ -d "$m" ]; then
meta_dir=`echo $m | sed 's%/%%'`
fi
doneif [ -z "$meta_dir" ]; then
meta_dir=".meta"
fi
"""
which means that if a "patches" directory exists it will use it since the command looks for untracked directories. I also noticed that some places define the default as ".meta", some as "meta" and they look a bit confused but that is probably a separate issue.
The question is then how does a "patches" directory end up in the kernel source. I was able to create one with the commands:
MACHINE=qemuppc bitbake linux-yocto perf -c clean MACHINE=qemuppc bitbake linux-yocto -c patch MACHINE=qemuppc bitbake perf -c unpack MACHINE=qemuppc bitbake linux-yocto -c kernel_configme
which doesn't fail like the autobuilder but does put the metadata into the wrong place with the wrong data (into patches). I'm therefore guessing this is a big horrible race.
Why does perf -c unpack create a patches directory? base.bbclass has:
do_unpack[cleandirs] = "${S}/patches"
The fix is therefore probably to not run the fetch/unpack/patch tasks in kernelsrc.bbclass.
Cheers,
Richard
Quoted 8 lines · Richard Purdie
Quoted 7 lines · Richard Purdie
I want to give people a headsup that we're having problems merging changes at the moment. We've been doing our best but the number of things building up which are causing issues is overwheling our ability to fix and stablise the build. It wasn't helped that I took a long weekend's vacation last weekend. There are changes being made or tested to the autobuilder which also isn't helping.
Quoted 6 lines
Quoted 3 lines
The kernel series has several issues:
* a random failure in
do_kernel_configme[i]I think I have a handle on this. If you look at the autobuilder failure it says:
do you have a link to the actual failure ?
Quoted 18 lines
| [INFO] Configuring target/machine combo: "standard/qemuppc" | [INFO] collecting configs in patches/meta-series
and what concerns me is "patches/meta-series". I my local builds it says .meta/meta-series. Poking around kern-tools I see:
"""
# determine the meta directory name. The meta directory is at the top level # of the repository, and is untracked. meta_dir_options=`git ls-files -o --directory` for m in $meta_dir_options; do if [ -d "$m" ]; then meta_dir=`echo $m | sed 's%/%%'` fi doneif [ -z "$meta_dir" ]; then
Quoted 8 lines
meta_dir=".meta"fi
"""which means that if a "patches" directory exists it will use it since the command looks for untracked directories. I also noticed that some places define the default as ".meta", some as "meta" and they look a bit confused but that is probably a separate issue.
They are consistent .. both are supported, we migrated from 'meta' to .meta some time ago, but there are old trees that still have to build.
Quoted 13 lines
The question is then how does a "patches" directory end up in the kernel source. I was able to create one with the commands:
MACHINE=qemuppc bitbake linux-yocto perf -c clean MACHINE=qemuppc bitbake linux-yocto -c patch MACHINE=qemuppc bitbake perf -c unpack MACHINE=qemuppc bitbake linux-yocto -c
kernel_configmewhich doesn't fail like the autobuilder but does put the metadata into the wrong place with the wrong data (into patches). I'm therefore guessing this is a big horrible race.
Quoted 6 lines
Why does perf -c unpack create a patches directory? base.bbclass has:
do_unpack[cleandirs] = "${S}/patches"
The fix is therefore probably to not run the fetch/unpack/patch tasks in kernelsrc.bbclass.
Very likely. I'll have a look. I noticed about 6 months ago that a patches directory was being created .. even when it was never going to be used. I took steps to remove it before the linux-yocto builds continued, but apparently it is sneaking back in through other means :)
Bruce
Quoted 6 lines
Cheers,
Richard
Quoted 13 lines · Bruce Ashfield
Quoted 12 lines · Richard Purdie
Quoted 10 lines · Richard Purdie
I want to give people a headsup that we're having problems merging changes at the moment. We've been doing our best but the number of things building up which are causing issues is overwheling our ability to fix and stablise the build. It wasn't helped that I took a long weekend's vacation last weekend. There are changes being made or tested to the autobuilder which also isn't helping.
The kernel series has several issues:
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
* a random failure in
do_kernel_configme[i]I think I have a handle on this. If you look at the autobuilder failure it says:
do you have a link to the actual failure ?
Aw crap. I see it in your original email now. I'll have a look.
Bruce
Quoted 74 lines
Quoted 20 lines
| [INFO] Configuring target/machine combo: "standard/qemuppc" | [INFO] collecting configs in patches/meta-series
and what concerns me is "patches/meta-series". I my local builds it says .meta/meta-series. Poking around kern-tools I see:
"""
# determine the meta directory name. The meta directory is at the top level # of the repository, and is untracked. meta_dir_options=`git ls-files -o --directory` for m in $meta_dir_options; do if [ -d "$m" ]; then meta_dir=`echo $m | sed 's%/%%'` fi doneif [ -z "$meta_dir" ]; then
Quoted 8 lines
meta_dir=".meta"fi
"""which means that if a "patches" directory exists it will use it since the command looks for untracked directories. I also noticed that some places define the default as ".meta", some as "meta" and they look a bit confused but that is probably a separate issue.
They are consistent .. both are supported, we migrated from 'meta' to .meta some time ago, but there are old trees that still have to build.
Quoted 13 lines
The question is then how does a "patches" directory end up in the kernel source. I was able to create one with the commands:
MACHINE=qemuppc bitbake linux-yocto perf -c clean MACHINE=qemuppc bitbake linux-yocto -c patch MACHINE=qemuppc bitbake perf -c unpack MACHINE=qemuppc bitbake linux-yocto -c
kernel_configmewhich doesn't fail like the autobuilder but does put the metadata into the wrong place with the wrong data (into patches). I'm therefore guessing this is a big horrible race.
Quoted 6 lines
Why does perf -c unpack create a patches directory? base.bbclass has:
do_unpack[cleandirs] = "${S}/patches"
The fix is therefore probably to not run the fetch/unpack/patch tasks in kernelsrc.bbclass.
Very likely. I'll have a look. I noticed about 6 months ago that a patches directory was being created .. even when it was never going to be used. I took steps to remove it before the linux-yocto builds continued, but apparently it is sneaking back in through other means :)
Bruce
Quoted 6 lines
Cheers,
Richard
--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Signature · 2 lines
"Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end"
Quoted 9 lines · Bruce Ashfield
Quoted 8 lines · Richard Purdie
Quoted 7 lines · Richard Purdie
I want to give people a headsup that we're having problems merging changes at the moment. We've been doing our best but the number of things building up which are causing issues is overwheling our ability to fix and stablise the build. It wasn't helped that I took a long weekend's vacation last weekend. There are changes being made or tested to the autobuilder which also isn't helping.
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
The kernel series has several issues:
* a random failure in
do_kernel_configme[i]I think I have a handle on this. If you look at the autobuilder failure it says:
do you have a link to the actual failure ?
It was in a previous mail:
Quoted 7 lines
Quoted 2 lines
The fix is therefore probably to not run the fetch/unpack/patch tasks in kernelsrc.bbclass.
Very likely. I'll have a look. I noticed about 6 months ago that a patches directory was being created .. even when it was never going to be used. I took steps to remove it before the linux-yocto builds continued, but apparently it is sneaking back in through other means :)
kernelsrc.bbclass sets ${S} to point at the directory so other recipes can create it.
Cheers,
Richard
Quoted 26 lines · Richard Purdie
Quoted 9 lines · Bruce Ashfield
Quoted 8 lines · Richard Purdie
Quoted 7 lines · Richard Purdie
I want to give people a headsup that we're having problems merging changes at the moment. We've been doing our best but the number of things building up which are causing issues is overwheling our ability to fix and stablise the build. It wasn't helped that I took a long weekend's vacation last weekend. There are changes being made or tested to the autobuilder which also isn't helping.
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
The kernel series has several issues:
* a random failure in
do_kernel_configme[i]I think I have a handle on this. If you look at the autobuilder failure it says:
do you have a link to the actual failure ?
It was in a previous mail:
Quoted 3 lines
Quoted 2 lines
The fix is therefore probably to not run the fetch/unpack/patch tasks in kernelsrc.bbclass.
Quoted 7 lines
Quoted 4 lines
Very likely. I'll have a look. I noticed about 6 months ago that a patches directory was being created .. even when it was never going to be used. I took steps to remove it before the linux-yocto builds continued, but apparently it is sneaking back in through other means :)
kernelsrc.bbclass sets ${S} to point at the directory so other recipes can create it.
aha yes. I'm familiar with doing battle with 'patches' being created in the build
directory, it broke the detection of the directory that is used to hold meta data .. and cost me about a week in August fixing things up.
With your latest patch, which problems are still outstanding ? I can have a look at them.
Bruce
Quoted 11 lines
Cheers,
Richard
--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Signature · 2 lines
"Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end"
So where are we at?
Thanks to some great help from Ross, we have a number of patches merged and many of the issues in my last email have been addressed. I'm continuing to struggle with the kernel series. The last build on the autobuilder highlighted that:
* there are problems in boot-directdisk.bbclass (have a fix)
* there is a do_rootfs/do_package_qa race (have a fix)
* the report-error.bbclass tasks could crash (have a fix)
* the kernelmodule sanity tests were failing (have a fix)
* qemumips gdb is failing to compile, probably due to new kernel headers (no fix as yet)
* systemd sanity QA tests continue to fail on xorg and systemd-login (no fix as yet)
* there are continuing problems with linux-imx from meta-fsl-arm, I thought these were addressed but clearly not :(
Ideally I'd like to take some time off over the holidays but I can't see that happening until the patch queues are under some kind of control :(.
Cheers,
Richard
Quoted 18 lines · Richard Purdie
So where are we at?
Thanks to some great help from Ross, we have a number of patches merged and many of the issues in my last email have been addressed. I'm continuing to struggle with the kernel series. The last build on the autobuilder highlighted that:
* there are problems in boot-directdisk.bbclass (have a fix)
* there is a do_rootfs/do_package_qa race (have a fix)
* the report-error.bbclass tasks could crash (have a fix)
* the kernelmodule sanity tests were failing (have a fix)
* qemumips gdb is failing to compile, probably due to new kernel headers (no fix as yet)
* systemd sanity QA tests continue to fail on xorg and systemd-login (no fix as yet)
* there are continuing problems with linux-imx from meta-fsl-arm, I thought these were addressed but clearly not :(
Quoted 2 lines
Ideally I'd like to take some time off over the holidays but I can't see that happening until the patch queues are under some kind of control :(.
Let me know if there are one of these that you want me to take. I'm all for pitching in and getting everyone some down time.
Bruce
Quoted 6 lines
Cheers,
Richard
Quoted 1 line · Richard Purdie
So where are we at?
...
Quoted 2 lines
* there are continuing problems with linux-imx from meta-fsl-arm, I thought these were addressed but clearly not :(
I have addressed those:
commit 150b8fe5fb428f9d62b8d5e589ba50bcb64bdb6a (HEAD, yocto/master-next, pending)
Author: Otavio Salvador <[email protected]> Date: Sat Dec 20 17:11:24 2014 -0200
fsl-kernel-localversion.bbclass: Fix building using out-of-tree
Since OE-Core:df9975f4 (kernel: Fix non linux-yocto builds) the way the build of kernel is done has change. Now the source code is moved placed at STAGING_KERNEL_DIR and the build is done at ${WORKDIR}/build which is B.
This patch adapts the configure task prepending to cope with those changes.
This is under meta-fsl-arm master-next branch so people give it a go in AB and let me know about any other issue you have.
Signature · 3 lines
Otavio Salvador O.S. Systems http://www.ossystems.com.br http://code.ossystems.com.br Mobile: +55 (53) 9981-7854 Mobile: +1 (347) 903-9750
Quoted 21 lines · Otavio Salvador
Quoted 1 line · Richard Purdie
So where are we at?
...
Quoted 2 lines
* there are continuing problems with linux-imx from meta-fsl-arm, I thought these were addressed but clearly not :(
I have addressed those:
commit
150b8fe5fb428f9d62b8d5e589ba50bcb64bdb6a(HEAD, yocto/master-next, pending)Author: Otavio Salvador <[email protected]> Date: Sat Dec 20 17:11:24 2014 -0200fsl-kernel-localversion.bbclass: Fix building using out-of-tree
Since OE-Core:df9975f4 (kernel: Fix non linux-yocto builds) the way the build of kernel is done has change. Now the source code is moved placed at
STAGING_KERNEL_DIRand the build is done at ${WORKDIR}/build which is B.
Quoted 4 lines
This patch adapts the configure task prepending to cope with those changes.
Change-Id I9ea4254ffd56e4b7201daf9a90b52756a399c0f6Signed-off-by Otavio Salvador <[email protected]>
I had tweaked kernel.bbclass so you shouldn't need to do that, the system notices if a kernel is configured in ${S} and moves the configuration to ${B}. I also fixed some other race issues which were leading to the makefile problems on older autobuilders.
Quoted 2 lines
This is under meta-fsl-arm master-next branch so people give it a go in AB and let me know about any other issue you have.
Is the latest failure (fsl-arm built ok), I think this may be a parallel make race?
Cheers,
Richard
Quoted 19 lines · Bruce Ashfield
Quoted 18 lines · Richard Purdie
So where are we at?
Thanks to some great help from Ross, we have a number of patches merged and many of the issues in my last email have been addressed. I'm continuing to struggle with the kernel series. The last build on the autobuilder highlighted that:
* there are problems in boot-directdisk.bbclass (have a fix)
* there is a do_rootfs/do_package_qa race (have a fix)
* the report-error.bbclass tasks could crash (have a fix)
* the kernelmodule sanity tests were failing (have a fix)
* qemumips gdb is failing to compile, probably due to new kernel headers (no fix as yet)
* systemd sanity QA tests continue to fail on xorg and systemd-login (no fix as yet)
* there are continuing problems with linux-imx from meta-fsl-arm, I thought these were addressed but clearly not :(
Quoted 5 lines
Quoted 2 lines
Ideally I'd like to take some time off over the holidays but I can't see that happening until the patch queues are under some kind of control :(.
Let me know if there are one of these that you want me to take. I'm all for pitching in and getting everyone some down time.
Thanks, I think with luck this is now nearly under control.
The systemd issue remains but I'm going to let someone else tackle that, its not from any recent patches.
There was another fsl-arm linux-imx failure which I've mentioned to Otavio. It looks like a parallel make race issue to me, probably specific to the imx kernel.
I'm still waiting on the autobuilder but things are looking a *lot* greener.
Cheers,
Richard
Quoted 19 lines · Bruce Ashfield
Quoted 18 lines · Richard Purdie
So where are we at?
Thanks to some great help from Ross, we have a number of patches merged and many of the issues in my last email have been addressed. I'm continuing to struggle with the kernel series. The last build on the autobuilder highlighted that:
* there are problems in boot-directdisk.bbclass (have a fix)
* there is a do_rootfs/do_package_qa race (have a fix)
* the report-error.bbclass tasks could crash (have a fix)
* the kernelmodule sanity tests were failing (have a fix)
* qemumips gdb is failing to compile, probably due to new kernel headers (no fix as yet)
* systemd sanity QA tests continue to fail on xorg and systemd-login (no fix as yet)
* there are continuing problems with linux-imx from meta-fsl-arm, I thought these were addressed but clearly not :(
Quoted 5 lines
Quoted 2 lines
Ideally I'd like to take some time off over the holidays but I can't see that happening until the patch queues are under some kind of control :(.
Let me know if there are one of these that you want me to take. I'm all for pitching in and getting everyone some down time.
Looks like I spoke too soon:
which seems to be a race over the kernel source directory. I'm guessing the file in question is a temporary build artefact of lttng-modules which was running at the same time? Any way we can avoid temp files in the kernel source dir?
Cheers,
Richard
Quoted 23 lines · Richard Purdie
Quoted 22 lines · Bruce Ashfield
Quoted 20 lines · Richard Purdie
So where are we at?
Thanks to some great help from Ross, we have a number of patches merged and many of the issues in my last email have been addressed. I'm continuing to struggle with the kernel series. The last build on the autobuilder highlighted that:
* there are problems in boot-directdisk.bbclass (have a fix)
* there is a do_rootfs/do_package_qa race (have a fix)
* the report-error.bbclass tasks could crash (have a fix)
* the kernelmodule sanity tests were failing (have a fix)
* qemumips gdb is failing to compile, probably due to new kernel headers (no fix as yet)
* systemd sanity QA tests continue to fail on xorg and systemd-login (no fix as yet)
* there are continuing problems with linux-imx from meta-fsl-arm, Ithought these were addressed but clearly not :(Ideally I'd like to take some time off over the holidays but I can't see that happening until the patch queues are under some kind of control :(.
Quoted 6 lines
Quoted 2 lines
Let me know if there are one of these that you want me to take. I'm all for pitching in and getting everyone some down time.
Looks like I spoke too soon:
Amazing how those race conditions pop out on the builders .. I probably built this 100 times, and never managed to trigger these.
Quoted 5 lines
which seems to be a race over the kernel source directory. I'm guessing the file in question is a temporary build artefact of lttng-modules which was running at the same time? Any way we can avoid temp files in the kernel source dir?
Hmm. The modules should already have their output directory set to their own source (not the kernel), so they really shouldn't be dropping down any temp files.
I was only using the cpio method to copy the source since that is what was already being used .. and it is a bit faster. The stat of the files to the pipe is what is catching us here.
The question is .. do we really care ? A straight copy of the files wouldn't be so sensitive to this, or we could explicitly exclude then in the existing cpio pipe. That would buy some time to track down what is touching down the tmp file in the kernel build process, and I can't see how the dual use of that staged directory will cause us a problem on the copied kernel source side (we do clean things up before packaging).
Bruce
Quoted 6 lines
Cheers,
Richard
Quoted 27 lines · Bruce Ashfield
Quoted 26 lines · Richard Purdie
Quoted 24 lines · Bruce Ashfield
Quoted 16 lines · Richard Purdie
So where are we at?
Thanks to some great help from Ross, we have a number of patches merged and many of the issues in my last email have been addressed. I'm continuing to struggle with the kernel series. The last build on the autobuilder highlighted that:
* there are problems in boot-directdisk.bbclass (have a fix)
* there is a do_rootfs/do_package_qa race (have a fix)
* the report-error.bbclass tasks could crash (have a fix)
* the kernelmodule sanity tests were failing (have a fix)
* qemumips gdb is failing to compile, probably due to new kernel headers (no fix as yet)
* systemd sanity QA tests continue to fail on xorg and systemd-login (no fix as yet)
* there are continuing problems with linux-imx from meta-fsl-arm, IQuoted 4 lines
thought these were addressed but clearly not :(Ideally I'd like to take some time off over the holidays but I can't see that happening until the patch queues are under some kind of control :(.
Let me know if there are one of these that you want me to take. I'm all for pitching in and getting everyone some down time.
Quoted 6 lines
Quoted 3 lines
Looks like I spoke too soon:
Amazing how those race conditions pop out on the builders .. I probably built this 100 times, and never managed to trigger these.
We also have one more:
On target module building for ppc fails.
Quoted 13 lines
Quoted 4 lines
which seems to be a race over the kernel source directory. I'm guessing the file in question is a temporary build artefact of lttng-modules which was running at the same time? Any way we can avoid temp files in the kernel source dir?
Hmm. The modules should already have their output directory set to their own source (not the kernel), so they really shouldn't be dropping down any temp files.
I was only using the cpio method to copy the source since that is what was already being used .. and it is a bit faster. The stat of the files to the pipe is what is catching us here.
Quoted 6 lines
The question is .. do we really care ? A straight copy of the files wouldn't be so sensitive to this, or we could explicitly exclude then in the existing cpio pipe. That would buy some time to track down what is touching down the tmp file in the kernel build process, and I can't see how the dual use of that staged directory will cause us a problem on the copied kernel source side (we do clean things up before packaging).
Well, I think we need to get to the bottom of this. I made some experiments. The best results were from hacking scripts/Kbuild.include where these .tmp files get created, I hacked it not to remove them.
When you do that, it becomes clear that do_make_scripts is the task which is causing the problem.
The world has changed since we last visited the scripts problem, we could move that task to the main kernel build now and have the modules and kernel-devsrc depend on it? The function in module.bbclass would then become a placeholder?
Cheers,
Richard
Quoted 65 lines · Richard Purdie
Quoted 27 lines · Bruce Ashfield
Quoted 26 lines · Richard Purdie
Quoted 24 lines · Bruce Ashfield
Quoted 16 lines · Richard Purdie
So where are we at?
Thanks to some great help from Ross, we have a number of patches merged and many of the issues in my last email have been addressed. I'm continuing to struggle with the kernel series. The last build on the autobuilder highlighted that:
* there are problems in boot-directdisk.bbclass (have a fix)
* there is a do_rootfs/do_package_qa race (have a fix)
* the report-error.bbclass tasks could crash (have a fix)
* the kernelmodule sanity tests were failing (have a fix)
* qemumips gdb is failing to compile, probably due to new kernel headers (no fix as yet)
* systemd sanity QA tests continue to fail on xorg and systemd-login (no fix as yet)
* there are continuing problems with linux-imx from meta-fsl-arm, IQuoted 4 lines
thought these were addressed but clearly not :(Ideally I'd like to take some time off over the holidays but I can't see that happening until the patch queues are under some kind of control :(.
Let me know if there are one of these that you want me to take. I'm all for pitching in and getting everyone some down time.
Quoted 6 lines
Quoted 3 lines
Looks like I spoke too soon:
Amazing how those race conditions pop out on the builders .. I probably built this 100 times, and never managed to trigger these.
We also have one more:
On target module building for ppc fails.
Quoted 13 lines
Quoted 4 lines
which seems to be a race over the kernel source directory. I'm guessing the file in question is a temporary build artefact of lttng-modules which was running at the same time? Any way we can avoid temp files in the kernel source dir?
Hmm. The modules should already have their output directory set to their own source (not the kernel), so they really shouldn't be dropping down any temp files.
I was only using the cpio method to copy the source since that is what was already being used .. and it is a bit faster. The stat of the files to the pipe is what is catching us here.
Quoted 6 lines
The question is .. do we really care ? A straight copy of the files wouldn't be so sensitive to this, or we could explicitly exclude then in the existing cpio pipe. That would buy some time to track down what is touching down the tmp file in the kernel build process, and I can't see how the dual use of that staged directory will cause us a problem on the copied kernel source side (we do clean things up before packaging).
Well, I think we need to get to the bottom of this. I made some experiments. The best results were from hacking
scripts/Kbuild.includewhere these .tmp files get created, I hacked it not to remove them.
Quoted 7 lines
When you do that, it becomes clear that
do_make_scriptsis the task which is causing the problem.The world has changed since we last visited the scripts problem, we could move that task to the main kernel build now and have the modules and kernel-devsrc depend on it? The function in module.bbclass would then become a placeholder?
After a lot of trail and error I've sent out a couple of patches for the above two issues. These fix the immediate problems however I think there is more cleanup than needs to be done early next year.
I filed:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=7095
since the overlap between kernel-dev and kernel-devsrc is problematic and should get resolved. This may help resolve the powerpc problem in a nicer way if devsrc is *just* the source and dev has some artefacts like the build config. It may be we have a separate package for the artefacts too, I'm open to ideas.
I did start to experiment with cleaning up the scripts handling in the build tree:
http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t222&id=760fa19a11165ff02c5df38cc89335fc503afba2
I have mixed feelings about it that, we really do need to try and do something about it though.
Cheers,
Richard
I thought we were over the hump and I merged the series. Next autobuild went rather red due to sstate cache usage. Reproducer was:
bitbake kernel-devsrc -c cleansstate
bitbake virtual/kernel
bitbake virtual/kernel -c clean
bitbake kernel-devsrc -c install
I've pushed yet another patch and restarted the build.
Cheers,
Richard
Quoted 61 lines · Richard Purdie
Quoted 27 lines · Bruce Ashfield
Quoted 26 lines · Richard Purdie
Quoted 24 lines · Bruce Ashfield
Quoted 16 lines · Richard Purdie
So where are we at?
Thanks to some great help from Ross, we have a number of patches merged and many of the issues in my last email have been addressed. I'm continuing to struggle with the kernel series. The last build on the autobuilder highlighted that:
* there are problems in boot-directdisk.bbclass (have a fix)
* there is a do_rootfs/do_package_qa race (have a fix)
* the report-error.bbclass tasks could crash (have a fix)
* the kernelmodule sanity tests were failing (have a fix)
* qemumips gdb is failing to compile, probably due to new kernel headers (no fix as yet)
* systemd sanity QA tests continue to fail on xorg and systemd-login (no fix as yet)
* there are continuing problems with linux-imx from meta-fsl-arm, IQuoted 4 lines
thought these were addressed but clearly not :(Ideally I'd like to take some time off over the holidays but I can't see that happening until the patch queues are under some kind of control :(.
Let me know if there are one of these that you want me to take. I'm all for pitching in and getting everyone some down time.
Quoted 6 lines
Quoted 3 lines
Looks like I spoke too soon:
Amazing how those race conditions pop out on the builders .. I probably built this 100 times, and never managed to trigger these.
We also have one more:
On target module building for ppc fails.
Quoted 13 lines
Quoted 4 lines
which seems to be a race over the kernel source directory. I'm guessing the file in question is a temporary build artefact of lttng-modules which was running at the same time? Any way we can avoid temp files in the kernel source dir?
Hmm. The modules should already have their output directory set to their own source (not the kernel), so they really shouldn't be dropping down any temp files.
I was only using the cpio method to copy the source since that is what was already being used .. and it is a bit faster. The stat of the files to the pipe is what is catching us here.
Quoted 6 lines
The question is .. do we really care ? A straight copy of the files wouldn't be so sensitive to this, or we could explicitly exclude then in the existing cpio pipe. That would buy some time to track down what is touching down the tmp file in the kernel build process, and I can't see how the dual use of that staged directory will cause us a problem on the copied kernel source side (we do clean things up before packaging).
Quoted 3 lines
Well, I think we need to get to the bottom of this. I made some experiments. The best results were from hacking
scripts/Kbuild.includewhere these .tmp files get created, I hacked it not to remove them.
back online after a 10 jaunt in the car.
I definitely agree that we needed to understand and fix this .. just that I would be away from my keyboard for about 24 hours, so putting something in place for the short term was the thought.
Quoted 8 lines
When you do that, it becomes clear that
do_make_scriptsis the task which is causing the problem.The world has changed since we last visited the scripts problem, we could move that task to the main kernel build now and have the modules and kernel-devsrc depend on it? The function in module.bbclass would then become a placeholder?
We can do that, and it makes sense there. Since both have a dependency on them being created .. moving it to a single place solves our issue and I don't see a bad side effect of the move.
Bruce
Quoted 12 lines
Cheers,
Richard
--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Signature · 2 lines
"Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end"
Quoted 69 lines · Richard Purdie
Quoted 68 lines · Richard Purdie
Quoted 27 lines · Bruce Ashfield
Quoted 26 lines · Richard Purdie
Quoted 24 lines · Bruce Ashfield
Quoted 16 lines · Richard Purdie
So where are we at?
Thanks to some great help from Ross, we have a number of patches merged and many of the issues in my last email have been addressed. I'm continuing to struggle with the kernel series. The last build on the autobuilder highlighted that:
* there are problems in boot-directdisk.bbclass (have a fix)
* there is a do_rootfs/do_package_qa race (have a fix)
* the report-error.bbclass tasks could crash (have a fix)
* the kernelmodule sanity tests were failing (have a fix)
* qemumips gdb is failing to compile, probably due to new kernel headers (no fix as yet)
* systemd sanity QA tests continue to fail on xorg and systemd-login (no fix as yet)
* there are continuing problems with linux-imx from meta-fsl-arm, IQuoted 4 lines
thought these were addressed but clearly not :(Ideally I'd like to take some time off over the holidays but I can't see that happening until the patch queues are under some kind of control :(.
Let me know if there are one of these that you want me to take. I'm all for pitching in and getting everyone some down time.
Quoted 6 lines
Quoted 3 lines
Looks like I spoke too soon:
Amazing how those race conditions pop out on the builders .. I probably built this 100 times, and never managed to trigger these.
We also have one more:
On target module building for ppc fails.
Quoted 13 lines
Quoted 4 lines
which seems to be a race over the kernel source directory. I'm guessing the file in question is a temporary build artefact of lttng-modules which was running at the same time? Any way we can avoid temp files in the kernel source dir?
Hmm. The modules should already have their output directory set to their own source (not the kernel), so they really shouldn't be dropping down any temp files.
I was only using the cpio method to copy the source since that is what was already being used .. and it is a bit faster. The stat of the files to the pipe is what is catching us here.
Quoted 6 lines
The question is .. do we really care ? A straight copy of the files wouldn't be so sensitive to this, or we could explicitly exclude then in the existing cpio pipe. That would buy some time to track down what is touching down the tmp file in the kernel build process, and I can't see how the dual use of that staged directory will cause us a problem on the copied kernel source side (we do clean things up before packaging).
Well, I think we need to get to the bottom of this. I made some experiments. The best results were from hacking
scripts/Kbuild.includewhere these .tmp files get created, I hacked it not to remove them.When you do that, it becomes clear that
do_make_scriptsis the task which is causing the problem.
Quoted 8 lines
Quoted 4 lines
The world has changed since we last visited the scripts problem, we could move that task to the main kernel build now and have the modules and kernel-devsrc depend on it? The function in module.bbclass would then become a placeholder?
After a lot of trail and error I've sent out a couple of patches for the above two issues. These fix the immediate problems however I think there is more cleanup than needs to be done early next year.
I replied out of order. I should have read this first. :)
Quoted 4 lines
I filed:
I'll grab that.
Quoted 6 lines
since the overlap between kernel-dev and kernel-devsrc is problematic and should get resolved. This may help resolve the powerpc problem in a nicer way if devsrc is *just* the source and dev has some artefacts like the build config. It may be we have a separate package for the artefacts too, I'm open to ideas.
It's a possibility. I'll throw together some ideas here as well.
Bruce
Quoted 24 lines
I did start to experiment with cleaning up the scripts handling in the build tree:
http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t222&id=760fa19a11165ff02c5df38cc89335fc503afba2I have mixed feelings about it that, we really do need to try and do something about it though.
Cheers,
Richard
--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Signature · 2 lines
"Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end"
Quoted 18 lines · Richard Purdie
So where are we at?
Thanks to some great help from Ross, we have a number of patches merged and many of the issues in my last email have been addressed. I'm continuing to struggle with the kernel series. The last build on the autobuilder highlighted that:
* there are problems in boot-directdisk.bbclass (have a fix)
* there is a do_rootfs/do_package_qa race (have a fix)
* the report-error.bbclass tasks could crash (have a fix)
* the kernelmodule sanity tests were failing (have a fix)
* qemumips gdb is failing to compile, probably due to new kernel headers (no fix as yet)
* systemd sanity QA tests continue to fail on xorg and systemd-login (no fix as yet)
* there are continuing problems with linux-imx from meta-fsl-arm, I thought these were addressed but clearly not :(
Quoted 2 lines
Ideally I'd like to take some time off over the holidays but I can't see that happening until the patch queues are under some kind of control :(.
One of kernel patches merged last weekend seems to be causing all /usr/src/kernel files to be installed in sysroot without sstate knowledge (and possibly missing when kernel is reused from sstate)
sstate-find-cruft.sh from last world build now reports:
ERROR: 52961 issues were found.
ERROR: Following files are installed in sysroot, but not tracked by sstate:
qemuarm/usr/src/kernel/.git qemuarm/usr/src/kernel/.git/HEAD
...
qemuarm/usr/src/kernel/.meta qemuarm/usr/src/kernel/.meta/cfg qemuarm/usr/src/kernel/.meta/cfg/kernel-cache qemuarm/usr/src/kernel/.meta/cfg/kernel-cache/00-README\
...
qemuarm/usr/src/kernel/.scmversion qemuarm/usr/src/kernel/COPYING qemuarm/usr/src/kernel/CREDITS qemuarm/usr/src/kernel/Documentation
...
Signature · 1 line
Martin 'JaMa' Jansa jabber: [email protected]
Quoted 19 lines · Martin Jansa
Quoted 18 lines · Richard Purdie
So where are we at?
Thanks to some great help from Ross, we have a number of patches merged and many of the issues in my last email have been addressed. I'm continuing to struggle with the kernel series. The last build on the autobuilder highlighted that:
* there are problems in boot-directdisk.bbclass (have a fix)
* there is a do_rootfs/do_package_qa race (have a fix)
* the report-error.bbclass tasks could crash (have a fix)
* the kernelmodule sanity tests were failing (have a fix)
* qemumips gdb is failing to compile, probably due to new kernel headers (no fix as yet)
* systemd sanity QA tests continue to fail on xorg and systemd-login (no fix as yet)
* there are continuing problems with linux-imx from meta-fsl-arm, I thought these were addressed but clearly not :(
Quoted 5 lines
Quoted 2 lines
Ideally I'd like to take some time off over the holidays but I can't see that happening until the patch queues are under some kind of control :(.
One of kernel patches merged last weekend seems to be causing all
/usr/src/kernelfiles to be installed in sysroot without sstate knowledge
That is intentional. The kernel is simply too large and the performance impact too great to put this under sstate control, we may as well just run the unpack again. usr/src/kernel should therefore be whitelisted.
Quoted 1 line
(and possibly missing when kernel is reused from sstate)
The system should be rerunning the kernel unpack process instead for any of these cases. If its not, let me know the situation and and we'll ensure the source does get re-extracted.
Cheers,
Richard