Skip to contentSkip to conversation

SetScene tasks hang forever?

DISCUSSION Rich Pixley 3 participants 19 messages May 2, 2012 – May 10, 2012

lore
Share & tools
lore

Rich Pixley

author
#0 lore raw

I'm seeing a lot of builds apparently hanging forever, (the ones that work seem to work within seconds - the ones that hang seem to hang for at least 10's of minutes), with:

rich@dolphin> nice tail -f Log

MACHINE           = "qemux86"
DISTRO            = ""
DISTRO_VERSION    = "oe-core.0"
TUNE_FEATURES     = "m32 i586"
TARGET_FPU        = ""
meta              = "master:35b5fb2dd2131d4c7dc6635c14c6e08ea6926457"

NOTE: Resolving any missing task queue dependencies
NOTE: Preparing runqueue
NOTE: Executing SetScene Tasks

If I run top, I see one processor pinned at 98 - 99% utilization running python, but no other clues.

Can anyone point me to doc, explain what's going on here, or point me in the right direction to debug this?

--rich

#1 lore raw
Quoted 16 lines · Rich Pixley

I'm seeing a lot of builds apparently hanging forever, (the ones that work seem to work within seconds - the ones that hang seem to hang for at least 10's of minutes), with:

rich@dolphin> nice tail -f Log

MACHINE           = "qemux86"
DISTRO            = ""
DISTRO_VERSION    = "oe-core.0"
TUNE_FEATURES     = "m32 i586"
TARGET_FPU        = ""
meta              = "master:35b5fb2dd2131d4c7dc6635c14c6e08ea6926457"

NOTE: Resolving any missing task queue dependencies
NOTE: Preparing runqueue
NOTE: Executing SetScene Tasks

Quoted 5 lines

If I run top, I see one processor pinned at 98 - 99% utilization running python, but no other clues.

Can anyone point me to doc, explain what's going on here, or point me in the right direction to debug this?

The only time I've seen "hang-like" behavior the system actually opened a devshell and was awaiting input. But based on your log, it doesn't look like that is the case.

Run bitbake with -DDD option, you will get considerably more debug information and it might help point out what it thinks it is doing.

--Mark

Quoted 6 lines

--rich

_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

author
#2 lore raw
Quoted 20 lines · Mark Hatle
Quoted 19 lines · Rich Pixley

I'm seeing a lot of builds apparently hanging forever, (the ones that work seem to work within seconds - the ones that hang seem to hang for at least 10's of minutes), with:

rich@dolphin> nice tail -f Log

MACHINE           = "qemux86"
DISTRO            = ""
DISTRO_VERSION    = "oe-core.0"
TUNE_FEATURES     = "m32 i586"
TARGET_FPU        = ""
meta              = "master:35b5fb2dd2131d4c7dc6635c14c6e08ea6926457"

NOTE: Resolving any missing task queue dependencies
NOTE: Preparing runqueue
NOTE: Executing SetScene Tasks

If I run top, I see one processor pinned at 98 - 99% utilization running python, but no other clues.

Quoted 8 lines
Quoted 2 lines

Can anyone point me to doc, explain what's going on here, or point me in the right direction to debug this?

The only time I've seen "hang-like" behavior the system actually opened a
devshell and was awaiting input. But based on your log, it doesn't look like
that is the case.

Run bitbake with -DDD option, you will get considerably more debug information and it might help point out what it thinks it is doing.

NOTE: Executing SetScene Tasks
DEBUG: Stamp for underlying task 12(virtual:native:/home/rich/projects/webos/openembedded-core/meta/recipes-devtools/opkg/opkg_svn.bb, do_populate_sysroot) is current, so skipping setscene variant
DEBUG: Stamp for underlying task 16(virtual:native:/home/rich/projects/webos/openembedded-core/meta/recipes-devtools/opkg-utils/opkg-utils_git.bb, do_populate_sysroot) is current, so skipping setscene variant
DEBUG: Stamp for underlying task 20(virtual:native:/home/rich/projects/webos/openembedded-core/meta/recipes-devtools/makedevs/makedevs_1.0.0.bb, do_populate_sysroot) is current, so skipping setscene variant
DEBUG: Stamp for underlying task 24(/home/rich/projects/webos/openembedded-core/meta/recipes-core/eglibc/ldconfig-native_2.12.1.bb, do_populate_sysroot) is current, so skipping setscene variant
DEBUG: Stamp for underlying task 32(virtual:native:/home/rich/projects/webos/openembedded-core/meta/recipes-devtools/genext2fs/genext2fs_1.4.1.bb, do_populate_sysroot) is current, so skipping setscene variant
DEBUG: Stamp for underlying task 36(virtual:native:/home/rich/projects/webos/openembedded-core/meta/recipes-devtools/e2fsprogs/e2fsprogs_1.42.1.bb, do_populate_sysroot) is current, so skipping setscene variant
DEBUG: Stamp for underlying task 40(virtual:native:/home/rich/projects/webos/openembedded-core/meta/recipes-devtools/qemu/qemu_0.15.1.bb, do_populate_sysroot) is current, so skipping setscene variant
DEBUG: Stamp for underlying task 44(/home/rich/projects/webos/openembedded-core/meta/recipes-devtools/qemu/qemu-helper-native_1.0.bb, do_populate_sysroot) is current, so skipping setscene variant

And then the spinning hang.

--rich

#3 lore raw
Quoted 55 lines · Rich Pixley
Quoted 20 lines · Mark Hatle
Quoted 19 lines · Rich Pixley

I'm seeing a lot of builds apparently hanging forever, (the ones that work seem to work within seconds - the ones that hang seem to hang for at least 10's of minutes), with:

rich@dolphin> nice tail -f Log

MACHINE           = "qemux86"
DISTRO            = ""
DISTRO_VERSION    = "oe-core.0"
TUNE_FEATURES     = "m32 i586"
TARGET_FPU        = ""
meta              = "master:35b5fb2dd2131d4c7dc6635c14c6e08ea6926457"

NOTE: Resolving any missing task queue dependencies
NOTE: Preparing runqueue
NOTE: Executing SetScene Tasks

If I run top, I see one processor pinned at 98 - 99% utilization running python, but no other clues.

Quoted 8 lines
Quoted 2 lines

Can anyone point me to doc, explain what's going on here, or point me in the right direction to debug this?

The only time I've seen "hang-like" behavior the system actually opened a
devshell and was awaiting input. But based on your log, it doesn't look like
that is the case.

Run bitbake with -DDD option, you will get considerably more debug information and it might help point out what it thinks it is doing.

NOTE: Executing SetScene Tasks
DEBUG: Stamp for underlying task 12(virtual:native:/home/rich/projects/webos/openembedded-core/meta/recipes-devtools/opkg/opkg_svn.bb, do_populate_sysroot) is current, so skipping setscene variant
DEBUG: Stamp for underlying task 16(virtual:native:/home/rich/projects/webos/openembedded-core/meta/recipes-devtools/opkg-utils/opkg-utils_git.bb, do_populate_sysroot) is current, so skipping setscene variant
DEBUG: Stamp for underlying task 20(virtual:native:/home/rich/projects/webos/openembedded-core/meta/recipes-devtools/makedevs/makedevs_1.0.0.bb, do_populate_sysroot) is current, so skipping setscene variant
DEBUG: Stamp for underlying task 24(/home/rich/projects/webos/openembedded-core/meta/recipes-core/eglibc/ldconfig-native_2.12.1.bb, do_populate_sysroot) is current, so skipping setscene variant
DEBUG: Stamp for underlying task 32(virtual:native:/home/rich/projects/webos/openembedded-core/meta/recipes-devtools/genext2fs/genext2fs_1.4.1.bb, do_populate_sysroot) is current, so skipping setscene variant
DEBUG: Stamp for underlying task 36(virtual:native:/home/rich/projects/webos/openembedded-core/meta/recipes-devtools/e2fsprogs/e2fsprogs_1.42.1.bb, do_populate_sysroot) is current, so skipping setscene variant
DEBUG: Stamp for underlying task 40(virtual:native:/home/rich/projects/webos/openembedded-core/meta/recipes-devtools/qemu/qemu_0.15.1.bb, do_populate_sysroot) is current, so skipping setscene variant
DEBUG: Stamp for underlying task 44(/home/rich/projects/webos/openembedded-core/meta/recipes-devtools/qemu/qemu-helper-native_1.0.bb, do_populate_sysroot) is current, so skipping setscene variant

Quoted 1 line

And then the spinning hang.

Sorry, I don't know how to continue debugging what might be wrong. The only other thing I can suggest is check that your filesystem is "real", not a netapp/nfs/network emulated filesystem....

And if you were continuing a previous build, start a new build directory and retry it.

--Mark

Quoted 6 lines

--rich

_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

author
#4 lore raw
Quoted 56 lines · Mark Hatle
Quoted 55 lines · Rich Pixley
Quoted 20 lines · Mark Hatle
Quoted 19 lines · Rich Pixley

I'm seeing a lot of builds apparently hanging forever, (the ones that work seem to work within seconds - the ones that hang seem to hang for at least 10's of minutes), with:

rich@dolphin>     nice tail -f Log
MACHINE           = "qemux86"
DISTRO            = ""
DISTRO_VERSION    = "oe-core.0"
TUNE_FEATURES     = "m32 i586"
TARGET_FPU        = ""
meta              = "master:35b5fb2dd2131d4c7dc6635c14c6e08ea6926457"

NOTE: Resolving any missing task queue dependencies
NOTE: Preparing runqueue
NOTE: Executing SetScene Tasks

If I run top, I see one processor pinned at 98 - 99% utilization running python, but no other clues.

Quoted 8 lines
Quoted 2 lines

Can anyone point me to doc, explain what's going on here, or point me in the right direction to debug this?

The only time I've seen "hang-like" behavior the system actually opened a
devshell and was awaiting input. But based on your log, it doesn't look like
that is the case.

Run bitbake with -DDD option, you will get considerably more debug information and it might help point out what it thinks it is doing.

NOTE: Executing SetScene Tasks
DEBUG: Stamp for underlying task 12(virtual:native:/home/rich/projects/webos/openembedded-core/meta/recipes-devtools/opkg/opkg_svn.bb, do_populate_sysroot) is current, so skipping setscene variant
DEBUG: Stamp for underlying task 16(virtual:native:/home/rich/projects/webos/openembedded-core/meta/recipes-devtools/opkg-utils/opkg-utils_git.bb, do_populate_sysroot) is current, so skipping setscene variant
DEBUG: Stamp for underlying task 20(virtual:native:/home/rich/projects/webos/openembedded-core/meta/recipes-devtools/makedevs/makedevs_1.0.0.bb, do_populate_sysroot) is current, so skipping setscene variant
DEBUG: Stamp for underlying task 24(/home/rich/projects/webos/openembedded-core/meta/recipes-core/eglibc/ldconfig-native_2.12.1.bb, do_populate_sysroot) is current, so skipping setscene variant
DEBUG: Stamp for underlying task 32(virtual:native:/home/rich/projects/webos/openembedded-core/meta/recipes-devtools/genext2fs/genext2fs_1.4.1.bb, do_populate_sysroot) is current, so skipping setscene variant
DEBUG: Stamp for underlying task 36(virtual:native:/home/rich/projects/webos/openembedded-core/meta/recipes-devtools/e2fsprogs/e2fsprogs_1.42.1.bb, do_populate_sysroot) is current, so skipping setscene variant
DEBUG: Stamp for underlying task 40(virtual:native:/home/rich/projects/webos/openembedded-core/meta/recipes-devtools/qemu/qemu_0.15.1.bb, do_populate_sysroot) is current, so skipping setscene variant
DEBUG: Stamp for underlying task 44(/home/rich/projects/webos/openembedded-core/meta/recipes-devtools/qemu/qemu-helper-native_1.0.bb, do_populate_sysroot) is current, so skipping setscene variant

Quoted 7 lines
Quoted 1 line

And then the spinning hang.

Sorry, I don't know how to continue debugging what might be wrong. The only other thing I can suggest is check that your filesystem is "real", not a netapp/nfs/network emulated filesystem....

And if you were continuing a previous build, start a new build directory and retry it.

Local file system. I'm building a second time expecting a null build pass. I was able to get a null build pass in the same directory yesterday.

Removing my build directory and starting over has been working, but costs me a few hours each time, and this happens frequently enough that I get no other work done. :(.

Thanks for reading.

--rich

#5 lore raw
Quoted 62 lines · Rich Pixley
Quoted 61 lines · Mark Hatle
Quoted 55 lines · Rich Pixley
Quoted 20 lines · Mark Hatle
Quoted 19 lines · Rich Pixley

I'm seeing a lot of builds apparently hanging forever, (the ones that work seem to work within seconds - the ones that hang seem to hang for at least 10's of minutes), with:

rich@dolphin> nice tail -f Log

MACHINE           = "qemux86"
DISTRO            = ""
DISTRO_VERSION    = "oe-core.0"
TUNE_FEATURES     = "m32 i586"
TARGET_FPU        = ""
meta              = "master:35b5fb2dd2131d4c7dc6635c14c6e08ea6926457"

NOTE: Resolving any missing task queue dependencies
NOTE: Preparing runqueue
NOTE: Executing SetScene Tasks

If I run top, I see one processor pinned at 98 - 99% utilization running python, but no other clues.

Quoted 8 lines
Quoted 2 lines

Can anyone point me to doc, explain what's going on here, or point me in the right direction to debug this?

The only time I've seen "hang-like" behavior the system actually opened a
devshell and was awaiting input. But based on your log, it doesn't look like
that is the case.

Run bitbake with -DDD option, you will get considerably more debug information and it might help point out what it thinks it is doing.

NOTE: Executing SetScene Tasks
DEBUG: Stamp for underlying task 12(virtual:native:/home/rich/projects/webos/openembedded-core/meta/recipes-devtools/opkg/opkg_svn.bb, do_populate_sysroot) is current, so skipping setscene variant
DEBUG: Stamp for underlying task 16(virtual:native:/home/rich/projects/webos/openembedded-core/meta/recipes-devtools/opkg-utils/opkg-utils_git.bb, do_populate_sysroot) is current, so skipping setscene variant
DEBUG: Stamp for underlying task 20(virtual:native:/home/rich/projects/webos/openembedded-core/meta/recipes-devtools/makedevs/makedevs_1.0.0.bb, do_populate_sysroot) is current, so skipping setscene variant
DEBUG: Stamp for underlying task 24(/home/rich/projects/webos/openembedded-core/meta/recipes-core/eglibc/ldconfig-native_2.12.1.bb, do_populate_sysroot) is current, so skipping setscene variant
DEBUG: Stamp for underlying task 32(virtual:native:/home/rich/projects/webos/openembedded-core/meta/recipes-devtools/genext2fs/genext2fs_1.4.1.bb, do_populate_sysroot) is current, so skipping setscene variant
DEBUG: Stamp for underlying task 36(virtual:native:/home/rich/projects/webos/openembedded-core/meta/recipes-devtools/e2fsprogs/e2fsprogs_1.42.1.bb, do_populate_sysroot) is current, so skipping setscene variant
DEBUG: Stamp for underlying task 40(virtual:native:/home/rich/projects/webos/openembedded-core/meta/recipes-devtools/qemu/qemu_0.15.1.bb, do_populate_sysroot) is current, so skipping setscene variant
DEBUG: Stamp for underlying task 44(/home/rich/projects/webos/openembedded-core/meta/recipes-devtools/qemu/qemu-helper-native_1.0.bb, do_populate_sysroot) is current, so skipping setscene variant

Quoted 1 line

And then the spinning hang.

Sorry, I don't know how to continue debugging what might be wrong. The only other thing I can suggest is check that your filesystem is "real", not a netapp/nfs/network emulated filesystem....

Quoted 8 lines
Quoted 2 lines

And if you were continuing a previous build, start a new build directory and retry it.

Local file system. I'm building a second time expecting a null build pass. I was able to get a null build pass in the same directory yesterday.

Removing my build directory and starting over has been working, but costs me a few hours each time, and this happens frequently enough that I get no other work done. :(.

Ya, that is certainly not acceptable. If you could file a bug on the bugzilla.yoctoproject.org someone might be able to help you diagnose this further and hopefully figure out a fix.

--Mark

Quoted 8 lines

Thanks for reading.

--rich

_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

#6 lore raw
Quoted 68 lines · Mark Hatle
Quoted 67 lines · Rich Pixley
Quoted 56 lines · Mark Hatle
Quoted 55 lines · Rich Pixley
Quoted 20 lines · Mark Hatle
Quoted 19 lines · Rich Pixley

I'm seeing a lot of builds apparently hanging forever, (the ones that work seem to work within seconds - the ones that hang seem to hang for at least 10's of minutes), with:

rich@dolphin> nice tail -f Log

MACHINE           = "qemux86"
DISTRO            = ""
DISTRO_VERSION    = "oe-core.0"
TUNE_FEATURES     = "m32 i586"
TARGET_FPU        = ""
meta              = "master:35b5fb2dd2131d4c7dc6635c14c6e08ea6926457"

NOTE: Resolving any missing task queue dependencies
NOTE: Preparing runqueue
NOTE: Executing SetScene Tasks

If I run top, I see one processor pinned at 98 - 99% utilization running python, but no other clues.

Quoted 8 lines
Quoted 2 lines

Can anyone point me to doc, explain what's going on here, or point me in the right direction to debug this?

The only time I've seen "hang-like" behavior the system actually opened a
devshell and was awaiting input. But based on your log, it doesn't look like
that is the case.

Run bitbake with -DDD option, you will get considerably more debug information and it might help point out what it thinks it is doing.

NOTE: Executing SetScene Tasks
DEBUG: Stamp for underlying task 12(virtual:native:/home/rich/projects/webos/openembedded-core/meta/recipes-devtools/opkg/opkg_svn.bb, do_populate_sysroot) is current, so skipping setscene variant
DEBUG: Stamp for underlying task 16(virtual:native:/home/rich/projects/webos/openembedded-core/meta/recipes-devtools/opkg-utils/opkg-utils_git.bb, do_populate_sysroot) is current, so skipping setscene variant
DEBUG: Stamp for underlying task 20(virtual:native:/home/rich/projects/webos/openembedded-core/meta/recipes-devtools/makedevs/makedevs_1.0.0.bb, do_populate_sysroot) is current, so skipping setscene variant
DEBUG: Stamp for underlying task 24(/home/rich/projects/webos/openembedded-core/meta/recipes-core/eglibc/ldconfig-native_2.12.1.bb, do_populate_sysroot) is current, so skipping setscene variant
DEBUG: Stamp for underlying task 32(virtual:native:/home/rich/projects/webos/openembedded-core/meta/recipes-devtools/genext2fs/genext2fs_1.4.1.bb, do_populate_sysroot) is current, so skipping setscene variant
DEBUG: Stamp for underlying task 36(virtual:native:/home/rich/projects/webos/openembedded-core/meta/recipes-devtools/e2fsprogs/e2fsprogs_1.42.1.bb, do_populate_sysroot) is current, so skipping setscene variant
DEBUG: Stamp for underlying task 40(virtual:native:/home/rich/projects/webos/openembedded-core/meta/recipes-devtools/qemu/qemu_0.15.1.bb, do_populate_sysroot) is current, so skipping setscene variant
DEBUG: Stamp for underlying task 44(/home/rich/projects/webos/openembedded-core/meta/recipes-devtools/qemu/qemu-helper-native_1.0.bb, do_populate_sysroot) is current, so skipping setscene variant

Quoted 7 lines
Quoted 1 line

And then the spinning hang.

Sorry, I don't know how to continue debugging what might be wrong. The only other thing I can suggest is check that your filesystem is "real", not a netapp/nfs/network emulated filesystem....

And if you were continuing a previous build, start a new build directory and retry it.

Local file system. I'm building a second time expecting a null build pass. I was able to get a null build pass in the same directory yesterday.

Quoted 7 lines
Quoted 3 lines

Removing my build directory and starting over has been working, but costs me a few hours each time, and this happens frequently enough that I get no other work done. :(.

Ya, that is certainly not acceptable. If you could file a bug on the bugzilla.yoctoproject.org someone might be able to help you diagnose this further and hopefully figure out a fix.

What would really help is a way to reproduce this...

Does it reproduce with a certain set of metadata/sstate perhaps?

What is odd about the above logs is that it appears bitbake never executes any task. Its possible something might have crashed somewhere I guess and not realise part of the system had died. Or it could be some kind of circular dependency loop where X needs Y to build and Y needs X so nothing happens. We are supposed to spot and error if that would have happened.

Does strace give an idea of which bits of bitbake are alive/looping? I'd probably resort to a few print()/bb.error() in the code at this point to find out what is alive, what is dead and where its looping...

Cheers,

Richard

author
#7 lore raw
Quoted 86 lines · Richard Purdie
Quoted 68 lines · Mark Hatle
Quoted 67 lines · Rich Pixley
Quoted 56 lines · Mark Hatle
Quoted 55 lines · Rich Pixley
Quoted 20 lines · Mark Hatle
Quoted 19 lines · Rich Pixley

I'm seeing a lot of builds apparently hanging forever, (the ones that work seem to work within seconds - the ones that hang seem to hang for at least 10's of minutes), with:

rich@dolphin> nice tail -f Log

MACHINE           = "qemux86"
DISTRO            = ""
DISTRO_VERSION    = "oe-core.0"
TUNE_FEATURES     = "m32 i586"
TARGET_FPU        = ""
meta              = "master:35b5fb2dd2131d4c7dc6635c14c6e08ea6926457"

NOTE: Resolving any missing task queue dependencies
NOTE: Preparing runqueue
NOTE: Executing SetScene Tasks

If I run top, I see one processor pinned at 98 - 99% utilization running python, but no other clues.

Quoted 8 lines
Quoted 2 lines

Can anyone point me to doc, explain what's going on here, or point me in the right direction to debug this?

The only time I've seen "hang-like" behavior the system actually opened a
devshell and was awaiting input. But based on your log, it doesn't look like
that is the case.

Run bitbake with -DDD option, you will get considerably more debug information and it might help point out what it thinks it is doing.

NOTE: Executing SetScene Tasks
DEBUG: Stamp for underlying task 12(virtual:native:/home/rich/projects/webos/openembedded-core/meta/recipes-devtools/opkg/opkg_svn.bb, do_populate_sysroot) is current, so skipping setscene variant
DEBUG: Stamp for underlying task 16(virtual:native:/home/rich/projects/webos/openembedded-core/meta/recipes-devtools/opkg-utils/opkg-utils_git.bb, do_populate_sysroot) is current, so skipping setscene variant
DEBUG: Stamp for underlying task 20(virtual:native:/home/rich/projects/webos/openembedded-core/meta/recipes-devtools/makedevs/makedevs_1.0.0.bb, do_populate_sysroot) is current, so skipping setscene variant
DEBUG: Stamp for underlying task 24(/home/rich/projects/webos/openembedded-core/meta/recipes-core/eglibc/ldconfig-native_2.12.1.bb, do_populate_sysroot) is current, so skipping setscene variant
DEBUG: Stamp for underlying task 32(virtual:native:/home/rich/projects/webos/openembedded-core/meta/recipes-devtools/genext2fs/genext2fs_1.4.1.bb, do_populate_sysroot) is current, so skipping setscene variant
DEBUG: Stamp for underlying task 36(virtual:native:/home/rich/projects/webos/openembedded-core/meta/recipes-devtools/e2fsprogs/e2fsprogs_1.42.1.bb, do_populate_sysroot) is current, so skipping setscene variant
DEBUG: Stamp for underlying task 40(virtual:native:/home/rich/projects/webos/openembedded-core/meta/recipes-devtools/qemu/qemu_0.15.1.bb, do_populate_sysroot) is current, so skipping setscene variant
DEBUG: Stamp for underlying task 44(/home/rich/projects/webos/openembedded-core/meta/recipes-devtools/qemu/qemu-helper-native_1.0.bb, do_populate_sysroot) is current, so skipping setscene variant

Quoted 7 lines
Quoted 1 line

And then the spinning hang.

Sorry, I don't know how to continue debugging what might be wrong. The only other thing I can suggest is check that your filesystem is "real", not a netapp/nfs/network emulated filesystem....

And if you were continuing a previous build, start a new build directory and retry it.

Local file system. I'm building a second time expecting a null build pass. I was able to get a null build pass in the same directory yesterday.

Quoted 6 lines
Quoted 3 lines

Removing my build directory and starting over has been working, but costs me a few hours each time, and this happens frequently enough that I get no other work done. :(.

Ya, that is certainly not acceptable. If you could file a bug on the bugzilla.yoctoproject.org someone might be able to help you diagnose this further and hopefully figure out a fix.

What would really help is a way to reproduce this...

Does it reproduce with a certain set of metadata/sstate perhaps?

What is odd about the above logs is that it appears bitbake never executes any task. Its possible something might have crashed somewhere I guess and not realise part of the system had died. Or it could be some kind of circular dependency loop where X needs Y to build and Y needs X so nothing happens. We are supposed to spot and error if that would have happened.

Quoted 3 lines

Does strace give an idea of which bits of bitbake are alive/looping? I'd probably resort to a few print()/bb.error() in the code at this point to find out what is alive, what is dead and where its looping...

I have more info now.

What I suspected was looping, (since it took longer than the ~1hr I was willing to wait), isn't actual looping. Given enough time, the builds do complete and I have comparable results on 5 different servers, (all ubuntu-12.04 amd64 and all on btrfs).

My initial, full builds of core-image-minimal do build, and they build in ~60min, (~30min if I hand seed the downloads directory). I'm using no mirrors other than the defaults. My second build in an already built directory, (expected to do nothing), takes anywhere from 7 - 10.5hrs to complete and successfully do nothing, depending on the server.

During this time, top shows a single cpu pinned at 98 - 100% utilization, and strace shows literally millions of access and stat calls on stamp files, mkdir on the stamps directory, etc. Statistical analysis of just the do_fetch access calls shows a distribution that seems to mimic the topological tree. That is, the most called access is for quilt-native and the components higher up the tree get fewer stats.

Oh, and the setscene stamps are all nonexistent. I presume that's expected.

First, I can't imagine why there would need to be more than one mkdir on the stamps directory within a single instantiation of bitbake. I can imagine that it was easier to attempt to mkdir it than to check first, but once it has been mkdir'd, (or checked), there's no need to do it another million times, is there?

Second, I can't imagine why there would need to be all the redundant stamp checking. That info is cached internally, isn't it?

And third, the fact that it seems to be checking the entire subtree what appear to be multiple times at every node suggests to me that the checking algorithm is broken. Back of the envelope... perhaps 300 components, maybe 10 tasks per component ~= 3e3 tasks. Figure a geometric explosion of checks for an inefficient algorithm and we're up to around 10e6 checks. I haven't counted an entire run, but based on the time it takes to run, I'd say I'm seeing one, maybe two orders of magnitude more checks than that. I've seen a few million node traversals in about 15min and a node traversal appears to involve several accesses and at least one stat.

I'm not familiar with the current bitbake internals so my next thought would be to replace the calls to access, stat, and mkdir on the stamp files with caching, counting calls. Build a dictionary of each file called, if it's new, do the kernel call and cache the result in the dictionary. If it's already in the dictionary, then inc a counter for it and return the cached value. This should a) improve the speed of the current algorithm, b) improve the speed of the eventual replacement algorithm, and c) give us some useful statistical data in the mean time.

I'm also going to try reformating one of the systems and compare how long a build on ext4 takes.

Any other ideas?

--rich

author
#8 lore raw
Quoted 141 lines · Rich Pixley
Quoted 86 lines · Richard Purdie
Quoted 68 lines · Mark Hatle
Quoted 67 lines · Rich Pixley
Quoted 56 lines · Mark Hatle
Quoted 55 lines · Rich Pixley
Quoted 20 lines · Mark Hatle
Quoted 19 lines · Rich Pixley

I'm seeing a lot of builds apparently hanging forever, (the ones that work seem to work within seconds - the ones that hang seem to hang for at least 10's of minutes), with:

rich@dolphin> nice tail -f Log

MACHINE           = "qemux86"
DISTRO            = ""
DISTRO_VERSION    = "oe-core.0"
TUNE_FEATURES     = "m32 i586"
TARGET_FPU        = ""
meta              = "master:35b5fb2dd2131d4c7dc6635c14c6e08ea6926457"

NOTE: Resolving any missing task queue dependencies
NOTE: Preparing runqueue
NOTE: Executing SetScene Tasks

If I run top, I see one processor pinned at 98 - 99% utilization running python, but no other clues.

Quoted 8 lines
Quoted 2 lines

Can anyone point me to doc, explain what's going on here, or point me in the right direction to debug this?

The only time I've seen "hang-like" behavior the system actually opened a
devshell and was awaiting input. But based on your log, it doesn't look like
that is the case.

Run bitbake with -DDD option, you will get considerably more debug information and it might help point out what it thinks it is doing.

NOTE: Executing SetScene Tasks
DEBUG: Stamp for underlying task 12(virtual:native:/home/rich/projects/webos/openembedded-core/meta/recipes-devtools/opkg/opkg_svn.bb, do_populate_sysroot) is current, so skipping setscene variant
DEBUG: Stamp for underlying task 16(virtual:native:/home/rich/projects/webos/openembedded-core/meta/recipes-devtools/opkg-utils/opkg-utils_git.bb, do_populate_sysroot) is current, so skipping setscene variant
DEBUG: Stamp for underlying task 20(virtual:native:/home/rich/projects/webos/openembedded-core/meta/recipes-devtools/makedevs/makedevs_1.0.0.bb, do_populate_sysroot) is current, so skipping setscene variant
DEBUG: Stamp for underlying task 24(/home/rich/projects/webos/openembedded-core/meta/recipes-core/eglibc/ldconfig-native_2.12.1.bb, do_populate_sysroot) is current, so skipping setscene variant
DEBUG: Stamp for underlying task 32(virtual:native:/home/rich/projects/webos/openembedded-core/meta/recipes-devtools/genext2fs/genext2fs_1.4.1.bb, do_populate_sysroot) is current, so skipping setscene variant
DEBUG: Stamp for underlying task 36(virtual:native:/home/rich/projects/webos/openembedded-core/meta/recipes-devtools/e2fsprogs/e2fsprogs_1.42.1.bb, do_populate_sysroot) is current, so skipping setscene variant
DEBUG: Stamp for underlying task 40(virtual:native:/home/rich/projects/webos/openembedded-core/meta/recipes-devtools/qemu/qemu_0.15.1.bb, do_populate_sysroot) is current, so skipping setscene variant
DEBUG: Stamp for underlying task 44(/home/rich/projects/webos/openembedded-core/meta/recipes-devtools/qemu/qemu-helper-native_1.0.bb, do_populate_sysroot) is current, so skipping setscene variant

Quoted 7 lines
Quoted 1 line

And then the spinning hang.

Sorry, I don't know how to continue debugging what might be wrong. The only other thing I can suggest is check that your filesystem is "real", not a netapp/nfs/network emulated filesystem....

And if you were continuing a previous build, start a new build directory and retry it.

Local file system. I'm building a second time expecting a null build pass. I was able to get a null build pass in the same directory yesterday.

Quoted 6 lines
Quoted 3 lines

Removing my build directory and starting over has been working, but costs me a few hours each time, and this happens frequently enough that I get no other work done. :(.

Ya, that is certainly not acceptable. If you could file a bug on the bugzilla.yoctoproject.org someone might be able to help you diagnose this further and hopefully figure out a fix.

What would really help is a way to reproduce this...

Does it reproduce with a certain set of metadata/sstate perhaps?

What is odd about the above logs is that it appears bitbake never executes any task. Its possible something might have crashed somewhere I guess and not realise part of the system had died. Or it could be some kind of circular dependency loop where X needs Y to build and Y needs X so nothing happens. We are supposed to spot and error if that would have happened.

Quoted 3 lines

Does strace give an idea of which bits of bitbake are alive/looping? I'd probably resort to a few print()/bb.error() in the code at this point to find out what is alive, what is dead and where its looping...

I have more info now.

What I suspected was looping, (since it took longer than the ~1hr I was willing to wait), isn't actual looping. Given enough time, the builds do complete and I have comparable results on 5 different servers, (all ubuntu-12.04 amd64 and all on btrfs).

My initial, full builds of core-image-minimal do build, and they build in ~60min, (~30min if I hand seed the downloads directory). I'm using no mirrors other than the defaults. My second build in an already built directory, (expected to do nothing), takes anywhere from 7 - 10.5hrs to complete and successfully do nothing, depending on the server.

During this time, top shows a single cpu pinned at 98 - 100% utilization, and strace shows literally millions of access and stat calls on stamp files, mkdir on the stamps directory, etc. Statistical analysis of just the do_fetch access calls shows a distribution that seems to mimic the topological tree. That is, the most called access is for quilt-native and the components higher up the tree get fewer stats.

Oh, and the setscene stamps are all nonexistent. I presume that's expected.

First, I can't imagine why there would need to be more than one mkdir on the stamps directory within a single instantiation of bitbake. I can imagine that it was easier to attempt to mkdir it than to check first, but once it has been mkdir'd, (or checked), there's no need to do it another million times, is there?

Second, I can't imagine why there would need to be all the redundant stamp checking. That info is cached internally, isn't it?

And third, the fact that it seems to be checking the entire subtree what appear to be multiple times at every node suggests to me that the checking algorithm is broken. Back of the envelope... perhaps 300 components, maybe 10 tasks per component ~= 3e3 tasks. Figure a geometric explosion of checks for an inefficient algorithm and we're up to around 10e6 checks. I haven't counted an entire run, but based on the time it takes to run, I'd say I'm seeing one, maybe two orders of magnitude more checks than that. I've seen a few million node traversals in about 15min and a node traversal appears to involve several accesses and at least one stat.

I'm not familiar with the current bitbake internals so my next thought would be to replace the calls to access, stat, and mkdir on the stamp files with caching, counting calls. Build a dictionary of each file called, if it's new, do the kernel call and cache the result in the dictionary. If it's already in the dictionary, then inc a counter for it and return the cached value. This should a) improve the speed of the current algorithm, b) improve the speed of the eventual replacement algorithm, and c) give us some useful statistical data in the mean time.

Quoted 2 lines

I'm also going to try reformating one of the systems and compare how long a build on ext4 takes.

A build on ext4 produces comparable results for me. ~30min initial build, ~7hrs for a second (do-nothing) build.

--rich

#9 lore raw
In reply to Rich Pixley
Quoted 68 lines · Rich Pixley
Quoted 13 lines · Richard Purdie
Quoted 1 line · Mark Hatle

On 5/2/12 2:45 PM, Rich Pixley wrote:

What would really help is a way to reproduce this...

Does it reproduce with a certain set of metadata/sstate perhaps?

What is odd about the above logs is that it appears bitbake never executes any task. Its possible something might have crashed somewhere I guess and not realise part of the system had died. Or it could be some kind of circular dependency loop where X needs Y to build and Y needs X so nothing happens. We are supposed to spot and error if that would have happened.

Quoted 3 lines

Does strace give an idea of which bits of bitbake are alive/looping? I'd probably resort to a few print()/bb.error() in the code at this point to find out what is alive, what is dead and where its looping...

I have more info now.

What I suspected was looping, (since it took longer than the ~1hr I was willing to wait), isn't actual looping. Given enough time, the builds do complete and I have comparable results on 5 different servers, (all ubuntu-12.04 amd64 and all on btrfs).

My initial, full builds of core-image-minimal do build, and they build in ~60min, (~30min if I hand seed the downloads directory). I'm using no mirrors other than the defaults. My second build in an already built directory, (expected to do nothing), takes anywhere from 7 - 10.5hrs to complete and successfully do nothing, depending on the server.

During this time, top shows a single cpu pinned at 98 - 100% utilization, and strace shows literally millions of access and stat calls on stamp files, mkdir on the stamps directory, etc. Statistical analysis of just the do_fetch access calls shows a distribution that seems to mimic the topological tree. That is, the most called access is for quilt-native and the components higher up the tree get fewer stats.

Oh, and the setscene stamps are all nonexistent. I presume that's expected.

First, I can't imagine why there would need to be more than one mkdir on the stamps directory within a single instantiation of bitbake. I can imagine that it was easier to attempt to mkdir it than to check first, but once it has been mkdir'd, (or checked), there's no need to do it another million times, is there?

Second, I can't imagine why there would need to be all the redundant stamp checking. That info is cached internally, isn't it?

And third, the fact that it seems to be checking the entire subtree what appear to be multiple times at every node suggests to me that the checking algorithm is broken. Back of the envelope... perhaps 300 components, maybe 10 tasks per component ~= 3e3 tasks. Figure a geometric explosion of checks for an inefficient algorithm and we're up to around 10e6 checks. I haven't counted an entire run, but based on the time it takes to run, I'd say I'm seeing one, maybe two orders of magnitude more checks than that. I've seen a few million node traversals in about 15min and a node traversal appears to involve several accesses and at least one stat.

I'm not familiar with the current bitbake internals so my next thought would be to replace the calls to access, stat, and mkdir on the stamp files with caching, counting calls. Build a dictionary of each file called, if it's new, do the kernel call and cache the result in the dictionary. If it's already in the dictionary, then inc a counter for it and return the cached value. This should a) improve the speed of the current algorithm, b) improve the speed of the eventual replacement algorithm, and c) give us some useful statistical data in the mean time.

Quoted 4 lines

I'm also going to try reformating one of the systems and compare how long a build on ext4 takes.

Any other ideas?

Well, this clearly doesn't happen with master or in any combination of the layers most users are using. The logical conclusion would be that there is something in your layer that is somehow triggering this.

Of course since that layer is secret and you can't show us it, we have a bit of a problem. Can you reproduce the bug against public code?

Are you by any chance setting BB_STAMP_POLICY somewhere?

Cheers,

Richard

author
#10 lore raw
Quoted 69 lines · Richard Purdie
Quoted 68 lines · Rich Pixley
Quoted 13 lines · Richard Purdie
Quoted 1 line · Mark Hatle

On 5/2/12 2:45 PM, Rich Pixley wrote:

What would really help is a way to reproduce this...

Does it reproduce with a certain set of metadata/sstate perhaps?

What is odd about the above logs is that it appears bitbake never executes any task. Its possible something might have crashed somewhere I guess and not realise part of the system had died. Or it could be some kind of circular dependency loop where X needs Y to build and Y needs X so nothing happens. We are supposed to spot and error if that would have happened.

Quoted 3 lines

Does strace give an idea of which bits of bitbake are alive/looping? I'd probably resort to a few print()/bb.error() in the code at this point to find out what is alive, what is dead and where its looping...

I have more info now.

What I suspected was looping, (since it took longer than the ~1hr I was willing to wait), isn't actual looping. Given enough time, the builds do complete and I have comparable results on 5 different servers, (all ubuntu-12.04 amd64 and all on btrfs).

My initial, full builds of core-image-minimal do build, and they build in ~60min, (~30min if I hand seed the downloads directory). I'm using no mirrors other than the defaults. My second build in an already built directory, (expected to do nothing), takes anywhere from 7 - 10.5hrs to complete and successfully do nothing, depending on the server.

During this time, top shows a single cpu pinned at 98 - 100% utilization, and strace shows literally millions of access and stat calls on stamp files, mkdir on the stamps directory, etc. Statistical analysis of just the do_fetch access calls shows a distribution that seems to mimic the topological tree. That is, the most called access is for quilt-native and the components higher up the tree get fewer stats.

Oh, and the setscene stamps are all nonexistent. I presume that's expected.

First, I can't imagine why there would need to be more than one mkdir on the stamps directory within a single instantiation of bitbake. I can imagine that it was easier to attempt to mkdir it than to check first, but once it has been mkdir'd, (or checked), there's no need to do it another million times, is there?

Second, I can't imagine why there would need to be all the redundant stamp checking. That info is cached internally, isn't it?

And third, the fact that it seems to be checking the entire subtree what appear to be multiple times at every node suggests to me that the checking algorithm is broken. Back of the envelope... perhaps 300 components, maybe 10 tasks per component ~= 3e3 tasks. Figure a geometric explosion of checks for an inefficient algorithm and we're up to around 10e6 checks. I haven't counted an entire run, but based on the time it takes to run, I'd say I'm seeing one, maybe two orders of magnitude more checks than that. I've seen a few million node traversals in about 15min and a node traversal appears to involve several accesses and at least one stat.

I'm not familiar with the current bitbake internals so my next thought would be to replace the calls to access, stat, and mkdir on the stamp files with caching, counting calls. Build a dictionary of each file called, if it's new, do the kernel call and cache the result in the dictionary. If it's already in the dictionary, then inc a counter for it and return the cached value. This should a) improve the speed of the current algorithm, b) improve the speed of the eventual replacement algorithm, and c) give us some useful statistical data in the mean time.

Quoted 7 lines
Quoted 4 lines

I'm also going to try reformating one of the systems and compare how long a build on ext4 takes.

Any other ideas?

Well, this clearly doesn't happen with master or in any combination of the layers most users are using. The logical conclusion would be that there is something in your layer that is somehow triggering this.

No private layer involved.

I do have a makefile which encapsulates the environment stuff, but that's it.

Quoted 2 lines

Of course since that layer is secret and you can't show us it, we have a bit of a problem. Can you reproduce the bug against public code?

Done. (Our layer is becoming open, we're committed to it, but it's a long process internally).

Quoted 1 line

Are you by any chance setting BB_STAMP_POLICY somewhere?

Yes. BB_STAMP_POLICY = "full".

I'll attach a copy of my local.conf and bblayers.conf.

--rich

# Time-stamp: <09-May-2012 10:50:03 PDT by [email protected]>

# Copyright (c) 2008 - 2012 Hewlett-Packard Development Company, L.P. #
# Licensed under the Apache License, Version 2.0 (the "License");

# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
##

# LAYER_CONF_VERSION is increased each time build/conf/bblayers.conf

# changes incompatibly
LCONF_VERSION = "4"

PALMDIR ?= "/home/rich/projects/webos"

OECORE_LAYER ?= "${PALMDIR}/openembedded-core/meta" WEBOS_LAYER ?= ""

BBFILES ?= ""
BBLAYERS ?= " \
${OECORE_LAYER} \
${WEBOS_LAYER} \
"

# DO NOT MODIFY! This script is generated by configure. Changes made

# here will be lost.  Source for this file is in local-conf.in.

# Time-stamp: <27-Apr-2012 15:23:26 PDT by [email protected]>

# Copyright (c) 2008 - 2012 Hewlett-Packard Development Company, L.P. #
# Licensed under the Apache License, Version 2.0 (the "License");

# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

MACHINE := "qemux86"

# Uncomment to have 'work' directories removed after a package builds #INHERIT += "rm_work"

BB_STAMP_POLICY = "full"
COVERAGE_BUILD = "0"
TMPDIR := "/home/rich/projects/webos/BUILD-qemux86" TCLIBCAPPEND := ""
PRODUCTION_BUILD := ""

# parallelization options
# there's an extra space in these CFLAGS such that defining
# 'TARGET_CFLAGS += ""' causes gdb to break.  I'm tired of looking for
# it for now.  Hence this strange construction of a naked trigger.
PARALLEL_MAKE := "-j 48"
BB_NUMBER_THREADS := "48"

BB_SRCREV_POLICY = "cache"
BB_FETCH_PREMIRRORONLY = "true"

# CONF_VERSION is increased each time build/conf/ changes incompatibly and is used to

# track the version of this file when it was generated. This can safely be ignored if
# this doesn't mean anything to you.
CONF_VERSION = "1"
#11 lore raw
Quoted 10 lines · Rich Pixley
Quoted 5 lines · Richard Purdie
Quoted 1 line · Rich Pixley

Any other ideas?

Well, this clearly doesn't happen with master or in any combination of the layers most users are using. The logical conclusion would be that there is something in your layer that is somehow triggering this.

No private layer involved.

I do have a makefile which encapsulates the environment stuff, but that's it.

Quoted 8 lines
Quoted 2 lines

Of course since that layer is secret and you can't show us it, we have a bit of a problem. Can you reproduce the bug against public code?

Done. (Our layer is becoming open, we're committed to it, but it's a long process internally).

Quoted 1 line

Are you by any chance setting BB_STAMP_POLICY somewhere?

Yes. BB_STAMP_POLICY = "full".

I'll attach a copy of my local.conf and bblayers.conf.

I'm 95% sure its BB_STAMP_POLICY = "full" causing the problems. The idea is really that sstate and other recent developments obsolete the "full" stamp code. I'm not sure it actually gets on with the setscene stamps the sstate code generates, as I suspect you're discovering.

We could try and fix the "full" policy, or we could just remove it. Looking at the code for the function that deals with this in runqueue.py, I can see where problems could occur.

So I guess I'm asking if we should fix that or can we remove it?

Cheers,

Richard

author
#12 lore raw
Quoted 24 lines · Richard Purdie
Quoted 10 lines · Rich Pixley
Quoted 5 lines · Richard Purdie
Quoted 1 line · Rich Pixley

Any other ideas?

Well, this clearly doesn't happen with master or in any combination of the layers most users are using. The logical conclusion would be that there is something in your layer that is somehow triggering this.

No private layer involved.

I do have a makefile which encapsulates the environment stuff, but that's it.

Quoted 8 lines
Quoted 2 lines

Of course since that layer is secret and you can't show us it, we have a bit of a problem. Can you reproduce the bug against public code?

Done. (Our layer is becoming open, we're committed to it, but it's a long process internally).

Quoted 1 line

Are you by any chance setting BB_STAMP_POLICY somewhere?

Yes. BB_STAMP_POLICY = "full".

I'll attach a copy of my local.conf and bblayers.conf.

I'm 95% sure its BB_STAMP_POLICY = "full" causing the problems. The idea is really that sstate and other recent developments obsolete the "full" stamp code. I'm not sure it actually gets on with the setscene stamps the sstate code generates, as I suspect you're discovering.

Quoted 5 lines

We could try and fix the "full" policy, or we could just remove it. Looking at the code for the function that deals with this in runqueue.py, I can see where problems could occur.

So I guess I'm asking if we should fix that or can we remove it?

Um... I'm not sure.

In the past, that was required to get everything built in an incremental fashion. That is, if A depended on B depended on C and C changed, BB_STAMP_POLICY = "full" was the only way to get A to be rebuilt automatically.

Are you saying that this happens automatically now even without the BB_STAMP_POLICY = "full" setting? Or that some other setting is more appropriate and perhaps has semantics I don't know? Or that the current default is no incremental rebuilds? Or... ?

--rich

#13 lore raw
Quoted 37 lines · Rich Pixley
Quoted 24 lines · Richard Purdie
Quoted 10 lines · Rich Pixley
Quoted 5 lines · Richard Purdie
Quoted 1 line · Rich Pixley

Any other ideas?

Well, this clearly doesn't happen with master or in any combination of the layers most users are using. The logical conclusion would be that there is something in your layer that is somehow triggering this.

No private layer involved.

I do have a makefile which encapsulates the environment stuff, but that's it.

Quoted 8 lines
Quoted 2 lines

Of course since that layer is secret and you can't show us it, we have a bit of a problem. Can you reproduce the bug against public code?

Done. (Our layer is becoming open, we're committed to it, but it's a long process internally).

Quoted 1 line

Are you by any chance setting BB_STAMP_POLICY somewhere?

Yes. BB_STAMP_POLICY = "full".

I'll attach a copy of my local.conf and bblayers.conf.

I'm 95% sure its BB_STAMP_POLICY = "full" causing the problems. The idea is really that sstate and other recent developments obsolete the "full" stamp code. I'm not sure it actually gets on with the setscene stamps the sstate code generates, as I suspect you're discovering.

Quoted 5 lines

We could try and fix the "full" policy, or we could just remove it. Looking at the code for the function that deals with this in runqueue.py, I can see where problems could occur.

So I guess I'm asking if we should fix that or can we remove it?

Um... I'm not sure.

In the past, that was required to get everything built in an incremental fashion. That is, if A depended on B depended on C and C changed, BB_STAMP_POLICY = "full" was the only way to get A to be rebuilt automatically.

Quoted 4 lines

Are you saying that this happens automatically now even without the BB_STAMP_POLICY = "full" setting? Or that some other setting is more appropriate and perhaps has semantics I don't know? Or that the current default is no incremental rebuilds? Or... ?

The settings that are now recommended are:

BB_SIGNATURE_HANDLER ?= 'OEBasicHash'
OELAYOUT_ABI = "8"

This requires a rebuild since the stamp file format changes, hence the ABI number increase. Currently, poky and angstrom use these settings amongst others but its not default in OE-Core. I'll likely propose a change to make it the default soon though.

This would then make BB_STAMP_POLICY = "full" obsolete and yet incremental builds will work correctly and likely rebuild less things (only really what potentially changed).

Cheers,

Richard

author
#14 lore raw
Quoted 5 lines · Richard Purdie

The settings that are now recommended are:

BB_SIGNATURE_HANDLER ?= 'OEBasicHash'
OELAYOUT_ABI = "8"

Quoted 8 lines

This requires a rebuild since the stamp file format changes, hence the ABI number increase. Currently, poky and angstrom use these settings amongst others but its not default in OE-Core. I'll likely propose a change to make it the default soon though.

This would then make BB_STAMP_POLICY = "full" obsolete and yet incremental builds will work correctly and likely rebuild less things (only really what potentially changed).

Thank you for the explanation.

I'll switch us immediately.

--rich

#15 lore raw
In reply to Rich Pixley

Hi Rich,

You might like to try the change below as I think it might address the problem.

Cheers,

Richard

bitbake/runqueue: Fix 'full' stamp checking to be more efficient and cache results

This should fix issues where bitbake would seemingly lock up when checking certain configurations of stampfiles.

Signed-off-by Richard Purdie <[email protected]>

---

bitbake/lib/bb/runqueue.py +7 -2
index b870caf..48433be 100644
@@ -875,7 +875,7 @@ class RunQueue:
             bb.msg.fatal("RunQueue", "check_stamps fatal internal error")
         return current
 
-    def check_stamp_task(self, task, taskname = None, recurse = False):
+    def check_stamp_task(self, task, taskname = None, recurse = False, cache = {}):
         def get_timestamp(f):
             try:
                 if not os.access(f, os.F_OK):

@@ -915,6 +915,9 @@ class RunQueue:
         t1 = get_timestamp(stampfile)
         for dep in self.rqdata.runq_depends[task]:
             if iscurrent:
+                if dep in cache:
+                    iscurrent = cache[dep]
+                    continue
                 fn2 = self.rqdata.taskData.fn_index[self.rqdata.runq_fnid[dep]]
                 taskname2 = self.rqdata.runq_task[dep]
                 stampfile2 = bb.build.stampfile(taskname2, self.rqdata.dataCache, fn2)

@@ -931,7 +934,9 @@ class RunQueue:
                         logger.debug(2, 'Stampfile %s < %s', stampfile, stampfile2)
                         iscurrent = False
                     if recurse and iscurrent:
-                        iscurrent = self.check_stamp_task(dep, recurse=True)
+                        iscurrent = self.check_stamp_task(dep, recurse=True, cache=cache)
+                        cache[dep] = iscurrent
+        cache[task] = iscurrent
         return iscurrent
 
     def execute_runqueue(self):
author
#16 lore raw

My rebuild completed in 5 seconds.

Thank you very much!

I'd rather not fork bitbake locally. Can I expect this patch to show up in git://git.openembedded.org/bitbake sometime soon?

--rich

Quoted 56 lines · Richard Purdie

Hi Rich,

You might like to try the change below as I think it might address the problem.

Cheers,

Richard

bitbake/runqueue: Fix 'full' stamp checking to be more efficient and cache results

This should fix issues where bitbake would seemingly lock up when checking certain configurations of stampfiles.

Signed-off-by Richard Purdie<[email protected]>

---

bitbake/lib/bb/runqueue.py +7 -2
index b870caf..48433be 100644
@@ -875,7 +875,7 @@ class RunQueue:
              bb.msg.fatal("RunQueue", "check_stamps fatal internal error")
          return current
 
-    def check_stamp_task(self, task, taskname = None, recurse = False):
+    def check_stamp_task(self, task, taskname = None, recurse = False, cache = {}):
          def get_timestamp(f):
              try:
                  if not os.access(f, os.F_OK):

@@ -915,6 +915,9 @@ class RunQueue:
          t1 = get_timestamp(stampfile)
          for dep in self.rqdata.runq_depends[task]:
              if iscurrent:
+                if dep in cache:
+                    iscurrent = cache[dep]
+                    continue
                  fn2 = self.rqdata.taskData.fn_index[self.rqdata.runq_fnid[dep]]
                  taskname2 = self.rqdata.runq_task[dep]
                  stampfile2 = bb.build.stampfile(taskname2, self.rqdata.dataCache, fn2)

@@ -931,7 +934,9 @@ class RunQueue:
                          logger.debug(2, 'Stampfile %s<  %s', stampfile, stampfile2)
                          iscurrent = False
                      if recurse and iscurrent:
-                        iscurrent = self.check_stamp_task(dep, recurse=True)
+                        iscurrent = self.check_stamp_task(dep, recurse=True, cache=cache)
+                        cache[dep] = iscurrent
+        cache[task] = iscurrent
          return iscurrent
 
      def execute_runqueue(self):

_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

#17 lore raw
Quoted 6 lines · Rich Pixley

My rebuild completed in 5 seconds.

Thank you very much!

I'd rather not fork bitbake locally. Can I expect this patch to show up in git://git.openembedded.org/bitbake sometime soon?

Yes, I just wanted to confirm we were fixing the right problem...

I'll post it to the bitbake list and merge in a day or two assuming no negative feedback.

I still wonder if that stamp mode is useful but that is a different discussion.

Cheers,

Richard

author
#18 lore raw
Quoted 11 lines · Richard Purdie
Quoted 6 lines · Rich Pixley

My rebuild completed in 5 seconds.

Thank you very much!

I'd rather not fork bitbake locally. Can I expect this patch to show up in git://git.openembedded.org/bitbake sometime soon?

Yes, I just wanted to confirm we were fixing the right problem...

I'll post it to the bitbake list and merge in a day or two assuming no negative feedback.

I'm running more thorough tests as I type.

Quoted 2 lines

I still wonder if that stamp mode is useful but that is a different discussion.

If I'm understanding your other message, then no, it's probably not.

My only concern is that in the A depends on B depends on C case, changing C forces a rebuild of A. I don't really care which mechanism makes that happen.

--rich