Skip to contentSkip to conversation

Fix weird rebuild issue even when sstate signature doesn't change

PATCH In review Kevin Tian 4 participants 14 messages Dec 14, 2010 – Dec 20, 2010

lore
Share & tools
lore

Kevin Tian

author
#0 lore raw

cover Fix weird rebuild issue even when sstate signature doesn't change

This patch is to fix one weird issue I've keeping seeing recently when using sstate. Even with a minimal build, there're around 50 recipes rebuilt from scratch even though sstate packages in sstate_cache do match. Actually the end result is to just overwrite sstate packages again with same sigature for those recipes.
https://lists.yoctoproject.org/pipermail/poky/2010-December/001063.html

The cause for this mess is from misinterpretation of the index of a list, which then points to wrong setscene tasks instead of desired ones. The end result is that some tasks which don't need execution are scheduled while other setscene tasks which need run are simply skipped.

It's based on Paul's sstate branch.

Pull URL: git://git.pokylinux.org/poky-contrib.git
Branch: tk/sstate
Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=tk/sstate

Thanks,
Kevin Tian <[email protected]>
---

Kevin Tian (1):
runqueue.py: use correct task ID when checking validity of setscene
tasks

 bitbake/lib/bb/runqueue.py |   16 +++++++++++-----
 1 files changed, 11 insertions(+), 5 deletions(-)
#1 lore raw
Quoted 6 lines · Kevin Tian

This patch is to fix one weird issue I've keeping seeing recently when using sstate. Even with a minimal build, there're around 50 recipes rebuilt from scratch even though sstate packages in sstate_cache do match. Actually the end result is to just overwrite sstate packages again with same sigature for those recipes.
https://lists.yoctoproject.org/pipermail/poky/2010-December/001063.html

Quoted 6 lines

The cause for this mess is from misinterpretation of the index of a list, which then points to wrong setscene tasks instead of desired ones. The end result is that some tasks which don't need execution are scheduled while other setscene tasks which need run are simply skipped.

It's based on Paul's sstate branch.

I've merged this patch into master, good catch and hopefully this gets sstate into a good working state.

I've merged an updated version of Paul's sstate branch too.

I'm hoping this makes sstate packages finally usable!

Cheers,

Richard

#2 lore raw
Quoted 15 lines

From: Richard Purdie [mailto:[email protected]]
Sent: Wednesday, December 15, 2010 5:12 PM

Quoted 11 lines · Kevin Tian

This patch is to fix one weird issue I've keeping seeing recently when using sstate. Even with a minimal build, there're around 50 recipes rebuilt from scratch even though sstate packages in sstate_cache do match. Actually the end result is to just overwrite sstate packages again with same sigature for those recipes.
https://lists.yoctoproject.org/pipermail/poky/2010-December/001063.html

The cause for this mess is from misinterpretation of the index of a list, which then points to wrong setscene tasks instead of desired ones. The end result is that some tasks which don't need execution are scheduled while other setscene tasks which need run are simply skipped.

Quoted 9 lines
Quoted 1 line

It's based on Paul's sstate branch.

I've merged this patch into master, good catch and hopefully this gets sstate into a good working state.

I've merged an updated version of Paul's sstate branch too.

I'm hoping this makes sstate packages finally usable!

I think now it's on much better shape. I tried minimal build with both same and different source dir. Only three packages are out of the scope:
linux-yocto
task-poky-boot
poky-image-minimal

poky-image-minimal is expected. task-poky-boot is caused by linux-yocto.do_deploy. The reason why linux-yocto.do_deploy fails to match is because it depends on DATETIME.

However there's one issue preventing me to come up a fix, which will be sent out in a separate thread for suggestion.

Thanks
Kevin

#3 lore raw
In reply to Richard Purdie
Quoted 12 lines · Richard Purdie
Quoted 11 lines · Kevin Tian

This patch is to fix one weird issue I've keeping seeing recently when using sstate. Even with a minimal build, there're around 50 recipes rebuilt from scratch even though sstate packages in sstate_cache do match. Actually the end result is to just overwrite sstate packages again with same sigature for those recipes.
https://lists.yoctoproject.org/pipermail/poky/2010-December/001063.html

The cause for this mess is from misinterpretation of the index of a list, which then points to wrong setscene tasks instead of desired ones. The end result is that some tasks which don't need execution are scheduled while other setscene tasks which need run are simply skipped.

Quoted 8 lines
Quoted 1 line

It's based on Paul's sstate branch.

I've merged this patch into master, good catch and hopefully this gets sstate into a good working state.

I've merged an updated version of Paul's sstate branch too.

I'm hoping this makes sstate packages finally usable!

Mucho bettero (TM) :-)

I just tried this from master and at least on the same machine but different build paths, it performed much as hoped. My initial build (step 1) took some 165 minutes, building the same target using the sstate cache from step 1 took only 22. The only packages rebuilt during step 2 were the kernel and the image tasks (I tested this with my own image+kernel recipes, so it's OK if this differs a little from other's results).

Bottom line, it seems to be working much better now.

Note to Paul: I also still see lots of LD_PRELOAD messages like
ERROR: ld.so: object 'libpseudo.so' from LD_PRELOAD cannot be preloaded: ignored.
but they did not seem to affect the result.

Note to Richard: I'm still seeing a ton of Noexec messages every time I rebuild a package in this tree. I thought I understood you to say they should happen at most once?

Signature · 4 lines
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------
#4 lore raw
Quoted 3 lines · On Wednesday 15 December 2010 16:10:16 you wrote:

Note to Paul: I also still see lots of LD_PRELOAD messages like
ERROR: ld.so: object 'libpseudo.so' from LD_PRELOAD cannot be preloaded: ignored.
but they did not seem to affect the result.

I think whether or not this causes breakage depends on BB_NUMBER_THREADS, the performance of your machine, etc. as it appears to be an order of operations issue.

Cheers,
Paul

#5 lore raw
In reply to Gary Thomas
Quoted 27 lines

From: Gary Thomas [mailto:[email protected]]
Sent: Thursday, December 16, 2010 12:10 AM

Quoted 12 lines · Richard Purdie
Quoted 11 lines · Kevin Tian

This patch is to fix one weird issue I've keeping seeing recently when using sstate. Even with a minimal build, there're around 50 recipes rebuilt from scratch even though sstate packages in sstate_cache do match. Actually the end result is to just overwrite sstate packages again with same sigature for those recipes.
https://lists.yoctoproject.org/pipermail/poky/2010-December/001063.html

The cause for this mess is from misinterpretation of the index of a list, which then points to wrong setscene tasks instead of desired ones. The end result is that some tasks which don't need execution are scheduled while other setscene tasks which need run are simply skipped.

Quoted 8 lines
Quoted 1 line

It's based on Paul's sstate branch.

I've merged this patch into master, good catch and hopefully this gets sstate into a good working state.

I've merged an updated version of Paul's sstate branch too.

I'm hoping this makes sstate packages finally usable!

Mucho bettero (TM) :-)

Quoted 7 lines

I just tried this from master and at least on the same machine but different build paths, it performed much as hoped. My initial build (step 1) took some 165 minutes, building the same target using the sstate cache from step 1 took only 22. The only packages rebuilt during step 2 were the kernel and the image tasks (I tested this with my own image+kernel recipes, so it's OK if this differs a little from other's results).

Good to see that. :-)

Thanks
Kevin

#6 lore raw
In reply to Gary Thomas
Quoted 7 lines

From: Gary Thomas [mailto:[email protected]]
Sent: Thursday, December 16, 2010 12:10 AM

Note to Richard: I'm still seeing a ton of Noexec messages every time I rebuild a package in this tree. I thought I understood you to say they should happen at most once?

Yes, RP's previous patch does reduce the Noexec messages a lot. However there remains one problem. Current stamp file check also calculates the stamp of tasks depended on. If those stamps are newer or not exist, the check return False to indicate mismatched stamp.

In 'noexec' case, some 'noexec' tasks happen to have some dependent tasks skipped which thus don't generate stamp files. This makes the check always failed for those 'noexec' tasks and thus you always observe the Noexec messages.

I have a patch to fix that, and will send out soon. :-)

Thanks
Kevin

#7 lore raw

about 'noexec' message

Quoted 16 lines

From: Tian, Kevin
Sent: Thursday, December 16, 2010 4:46 PM

Quoted 7 lines

From: Gary Thomas [mailto:[email protected]]
Sent: Thursday, December 16, 2010 12:10 AM

Note to Richard: I'm still seeing a ton of Noexec messages every time I rebuild a package in this tree. I thought I understood you to say they should happen at most once?

Yes, RP's previous patch does reduce the Noexec messages a lot. However there remains one problem. Current stamp file check also calculates the stamp of tasks depended on. If those stamps are newer or not exist, the check return False to indicate mismatched stamp.

Quoted 6 lines

In 'noexec' case, some 'noexec' tasks happen to have some dependent tasks skipped which thus don't generate stamp files. This makes the check always failed for those 'noexec' tasks and thus you always observe the Noexec messages.

I have a patch to fix that, and will send out soon. :-)

Though I managed to remove the rest 'noexec' messages by adding below in check_stamp_task():

if 'noexec' in taskdep and taskname in taskdep['noexec']
return True

I realized there's one potential risk. Though 'noexec' task is not executed at all, it's considered to be a normal task which normally plays the role of the sync points. Ideally if there're changes on stamp file of tasks that this 'noexec' task depends on, we still want to generate a new stamp file for the 'noexec' task. That way other tasks depending on that 'noexec' task will trigger rebuild too.

If we always take 'noexec' task away from the comparison, above chain effect disappears.

Current comparison condition for stamp files is:

if not t2 or t1 < t2
iscurrent = False

I'm not sure under which case a stamp file (t2) doesn't exist, and whether it's safe to use:
if not t2 and not noexec or t2 and t1 < t2:
iscurrent = False

to check stamp file for 'noexec' task.

Or alternative is to simply convert bb.note to bb.debug for 'noexec' message. :-)

Thanks
Kevin

#8 lore raw

Re: about 'noexec' message

Quoted 43 lines · Kevin
Quoted 16 lines

From: Tian, Kevin
Sent: Thursday, December 16, 2010 4:46 PM

Quoted 7 lines

From: Gary Thomas [mailto:[email protected]]
Sent: Thursday, December 16, 2010 12:10 AM

Note to Richard: I'm still seeing a ton of Noexec messages every time I rebuild a package in this tree. I thought I understood you to say they should happen at most once?

Yes, RP's previous patch does reduce the Noexec messages a lot. However there remains one problem. Current stamp file check also calculates the stamp of tasks depended on. If those stamps are newer or not exist, the check return False to indicate mismatched stamp.

Quoted 6 lines

In 'noexec' case, some 'noexec' tasks happen to have some dependent tasks skipped which thus don't generate stamp files. This makes the check always failed for those 'noexec' tasks and thus you always observe the Noexec messages.

I have a patch to fix that, and will send out soon. :-)

Though I managed to remove the rest 'noexec' messages by adding below in check_stamp_task():

if 'noexec' in taskdep and taskname in taskdep['noexec']
return True

I realized there's one potential risk. Though 'noexec' task is not executed at all, it's considered to be a normal task which normally plays the role of the sync points. Ideally if there're changes on stamp file of tasks that this 'noexec' task depends on, we still want to generate a new stamp file for the 'noexec' task. That way other tasks depending on that 'noexec' task will trigger rebuild too.

If we always take 'noexec' task away from the comparison, above chain effect disappears.

Current comparison condition for stamp files is:

if not t2 or t1 < t2
iscurrent = False

Quoted 8 lines

I'm not sure under which case a stamp file (t2) doesn't exist, and whether it's safe to use:
if not t2 and not noexec or t2 and t1 < t2:
iscurrent = False

to check stamp file for 'noexec' task.

Or alternative is to simply convert bb.note to bb.debug for 'noexec' message. :-)

I've looked at this and realised that scenequeue should not be trying to skip noexec tasks as this means the stamps don't get generated. Instead, it should just failoutright and let the secondary code in the main runqueue deal with them at the correct point in time.

I've pushed this change, lets see if it solves the issues people have been seeing. It certainly makes the output from bitbake more readable again.

Cheers,

Richard

#9 lore raw

Re: about 'noexec' message

Quoted 56 lines

From: Richard Purdie [mailto:[email protected]]
Sent: Thursday, December 16, 2010 11:19 PM

Quoted 43 lines · Kevin
Quoted 16 lines

From: Tian, Kevin
Sent: Thursday, December 16, 2010 4:46 PM

Quoted 7 lines

From: Gary Thomas [mailto:[email protected]]
Sent: Thursday, December 16, 2010 12:10 AM

Note to Richard: I'm still seeing a ton of Noexec messages every time I rebuild a package in this tree. I thought I understood you to say they should happen at most once?

Yes, RP's previous patch does reduce the Noexec messages a lot. However there remains one problem. Current stamp file check also calculates the stamp of tasks depended on. If those stamps are newer or not exist, the check return False to indicate mismatched stamp.

Quoted 6 lines

In 'noexec' case, some 'noexec' tasks happen to have some dependent tasks skipped which thus don't generate stamp files. This makes the check always failed for those 'noexec' tasks and thus you always observe the Noexec messages.

I have a patch to fix that, and will send out soon. :-)

Though I managed to remove the rest 'noexec' messages by adding below in check_stamp_task():

if 'noexec' in taskdep and taskname in taskdep['noexec']
return True

I realized there's one potential risk. Though 'noexec' task is not executed at all, it's considered to be a normal task which normally plays the role of the sync points. Ideally if there're changes on stamp file of tasks that this 'noexec' task depends on, we still want to generate a new stamp file for the 'noexec' task. That way other tasks depending on that 'noexec' task will trigger rebuild too.

If we always take 'noexec' task away from the comparison, above chain effect disappears.

Current comparison condition for stamp files is:

if not t2 or t1 < t2
iscurrent = False

Quoted 8 lines

I'm not sure under which case a stamp file (t2) doesn't exist, and whether it's safe to use:
if not t2 and not noexec or t2 and t1 < t2:
iscurrent = False

to check stamp file for 'noexec' task.

Or alternative is to simply convert bb.note to bb.debug for 'noexec' message. :-)

Quoted 9 lines

I've looked at this and realised that scenequeue should not be trying to skip noexec tasks as this means the stamps don't get generated. Instead, it should just failoutright and let the secondary code in the main runqueue deal with them at the correct point in time.

I've pushed this change, lets see if it solves the issues people have been seeing. It certainly makes the output from bitbake more readable again.

yes, this makes the output pretty clean now. thanks for the fix.

In the meantime I'm curious how your fix solves the issue, as I can't map your commit message to the change. Sorry that I may need your help here. :-)

It looks that with your fix those 'noexec' setscene tasks which are originally skipped now still enter the main runqueue logic to get their stamps file generated, and thus later when normal 'noexec' tasks are checked all dependent tasks have got their stamp files ready. Is it true?

However I didn't see much difference between task_failoutright and task_skip, which neither touches the stamp file. Could you shed some light how this impacts the main runqueue logic. I search the code, it looks that only when a task is actually scheduled to run ("exec_task") make_stamp() is invoked...

Last question for the check_stamp_task condition:

+ if not t2:
+ bb.msg.debug(2, bb.msg.domain.RunQueue, "Stampfile %s d
iscurrent = False
+ if t1 < t2:
+ bb.msg.debug(2, bb.msg.domain.RunQueue, "Stampfile %s <
+ iscurrent = False
+

Under which case a task depended by others may not get its stamp file ready?

Thanks
Kevin

#10 lore raw

Re: about 'noexec' message

In reply to Richard Purdie
Quoted 56 lines

From: Richard Purdie [mailto:[email protected]]
Sent: Thursday, December 16, 2010 11:19 PM

Quoted 43 lines · Kevin
Quoted 16 lines

From: Tian, Kevin
Sent: Thursday, December 16, 2010 4:46 PM

Quoted 7 lines

From: Gary Thomas [mailto:[email protected]]
Sent: Thursday, December 16, 2010 12:10 AM

Note to Richard: I'm still seeing a ton of Noexec messages every time I rebuild a package in this tree. I thought I understood you to say they should happen at most once?

Yes, RP's previous patch does reduce the Noexec messages a lot. However there remains one problem. Current stamp file check also calculates the stamp of tasks depended on. If those stamps are newer or not exist, the check return False to indicate mismatched stamp.

Quoted 6 lines

In 'noexec' case, some 'noexec' tasks happen to have some dependent tasks skipped which thus don't generate stamp files. This makes the check always failed for those 'noexec' tasks and thus you always observe the Noexec messages.

I have a patch to fix that, and will send out soon. :-)

Though I managed to remove the rest 'noexec' messages by adding below in check_stamp_task():

if 'noexec' in taskdep and taskname in taskdep['noexec']
return True

I realized there's one potential risk. Though 'noexec' task is not executed at all, it's considered to be a normal task which normally plays the role of the sync points. Ideally if there're changes on stamp file of tasks that this 'noexec' task depends on, we still want to generate a new stamp file for the 'noexec' task. That way other tasks depending on that 'noexec' task will trigger rebuild too.

If we always take 'noexec' task away from the comparison, above chain effect disappears.

Current comparison condition for stamp files is:

if not t2 or t1 < t2
iscurrent = False

Quoted 8 lines

I'm not sure under which case a stamp file (t2) doesn't exist, and whether it's safe to use:
if not t2 and not noexec or t2 and t1 < t2:
iscurrent = False

to check stamp file for 'noexec' task.

Or alternative is to simply convert bb.note to bb.debug for 'noexec' message. :-)

Quoted 9 lines

I've looked at this and realised that scenequeue should not be trying to skip noexec tasks as this means the stamps don't get generated. Instead, it should just failoutright and let the secondary code in the main runqueue deal with them at the correct point in time.

I've pushed this change, lets see if it solves the issues people have been seeing. It certainly makes the output from bitbake more readable again.

Unfortunately I think this causes another problem when using sstate. I got an error from m4-native when building from scratch (with prebuilt available):

ERROR: install: cannot stat `xxx/tmp/sysroots/i686-linux/usr/share/gnu-config/config.sub

Actually from the log gnu-config.do_populate_sysroot has been accelerated earlier in setscene stage, with necessary sysroot files installed. However later gnu-config-native is still considered requiring rebuild, and thus the installed sysroot files are removed again. The weird thing under this condition is that m4-native thought that do_populate_sysroot of gnu-config-native has been completed and thus doesn't wait for new build of gnu-config-native to complete, which then triggers above error.

I haven't figure out the reason why this commit causes this confliction: sstate package is installed successfully but the package still requires rebuild. By reverting this change this error disappears.

Thanks
Kevin

#11 lore raw

Re: about 'noexec' message

Quoted 12 lines · Kevin

Unfortunately I think this causes another problem when using sstate. I got an error from m4-native when building from scratch (with prebuilt available):

ERROR: install: cannot stat `xxx/tmp/sysroots/i686-linux/usr/share/gnu-config/config.sub

Actually from the log gnu-config.do_populate_sysroot has been accelerated earlier in setscene stage, with necessary sysroot files installed. However later gnu-config-native is still considered requiring rebuild, and thus the installed sysroot files are removed again. The weird thing under this condition is that m4-native thought that do_populate_sysroot of gnu-config-native has been completed and thus doesn't wait for new build of gnu-config-native to complete, which then triggers above error.

Quoted 3 lines

I haven't figure out the reason why this commit causes this confliction: sstate package is installed successfully but the package still requires rebuild. By reverting this change this error disappears.

I made some assumptions in the fix I pushed which weren't correct but didn't show up under the testing I gave it. The real underlying issue is that the XXX_setscene stamps are never created for setscene noexec tasks. I've pushed a fix which does this now:

http://git.pokylinux.org/cgit.cgi/poky/commit/?id=a86f960f651c561bb104723aa0b4cd0922a3425a

Cheers,

Richard

#12 lore raw
In reply to Tian, Kevin
Quoted 13 lines · Kevin
Quoted 7 lines

From: Gary Thomas [mailto:[email protected]]
Sent: Thursday, December 16, 2010 12:10 AM

Note to Richard: I'm still seeing a ton of Noexec messages every time I rebuild a package in this tree. I thought I understood you to say they should happen at most once?

Yes, RP's previous patch does reduce the Noexec messages a lot. However there remains one problem. Current stamp file check also calculates the stamp of tasks depended on. If those stamps are newer or not exist, the check return False to indicate mismatched stamp.

Quoted 5 lines

In 'noexec' case, some 'noexec' tasks happen to have some dependent tasks skipped which thus don't generate stamp files. This makes the check always failed for those 'noexec' tasks and thus you always observe the Noexec messages.

I have a patch to fix that, and will send out soon. :-)

I've pushed what I think is the correct fix for this problem, let me know how it looks.

Cheers,

Richard

#13 lore raw
In reply to Gary Thomas
Quoted 19 lines · Gary Thomas
Quoted 7 lines · Richard Purdie

On Tue, 2010-12-14 at 21:40 +0800, Kevin Tian wrote:
I've merged this patch into master, good catch and hopefully this gets sstate into a good working state.

I've merged an updated version of Paul's sstate branch too.

I'm hoping this makes sstate packages finally usable!

Mucho bettero (TM) :-)

I just tried this from master and at least on the same machine but different build paths, it performed much as hoped. My initial build (step 1) took some 165 minutes, building the same target using the sstate cache from step 1 took only 22. The only packages rebuilt during step 2 were the kernel and the image tasks (I tested this with my own image+kernel recipes, so it's OK if this differs a little from other's results).

Quoted 5 lines

Bottom line, it seems to be working much better now.

Note to Paul: I also still see lots of LD_PRELOAD messages like
ERROR: ld.so: object 'libpseudo.so' from LD_PRELOAD cannot be preloaded: ignored.
but they did not seem to affect the result.

This error should be fixed with some changes that went in today.

Quoted 3 lines

Note to Richard: I'm still seeing a ton of Noexec messages every time I rebuild a package in this tree. I thought I understood you to say they should happen at most once?

Yes, I've pushed some changes which should help too.

Its taken longer that I'd have liked but we *are* getting there with sstate and prebuilds! :)

Cheers,

Richard