RSS difficulties

I'm having a few problems adapting our build setup to work with recipe specific sysroots. More specifically, I am finding it difficult to get the correct set of components installed into the recipe-sysroot in all cases. In some cases I am getting too much (which causes build failures if oe attempts to stage two things that install the same file) and in other cases I am not getting enough.

Could somebody explain, ideally using only small words, how the set of things to populate in the recipe-sysroot is determined? I assume it is based on the task dependency graph somehow but I'm not quite sure which relationships I ought to be looking at.

thanks

p.

It comes down to what setscene_depvalid() in sstate.bbclass returns. If that returns False a given dependency is installed, it if returns True, it is skipped and not installed.

Reading that function should give some clues as to the logic it applies. It operates not on a full task graph but on version collapsed only to sstate tasks.

The logic should mirror what would get installed from sstate if that sstate were available, hence the logic is somewhat tested and not just new for RSS. That doesn't mean its right in all cases though!

If that doesn't help, perhaps some specific examples of what you think should be being installed and isn't or things which are being installed and shouldn't might help.

Cheers,

Richard

Ah, right, thanks. The particular piece of magic I was missing was:

            # Nothing need depend on libc-initial/gcc-cross-initial if "-initial" in taskdependees[task][0]:
                continue

which I think explains all the difficulties I was having.

p.

Actually, that explains most of my difficulties but not all of them. The remaining ones seem to come down to missing dependencies but I am not quite clear on why the dependencies are missing.

The problematic recipes are Java ones. The Java toolchain, as you might expect, has a fairly complicated dependency tree but essentially:

- any Java package has DEPENDS = "virtual/javac-native", which is PROVIDEd by ecj-bootstrap-native.bb

- ecj-bootstrap-native.bb is basically just a wrapper script which runs the Java compiler. The actual compiler implementation is in libecj- bootstrap-native.bb and ecj-bootstrap-native.bb RDEPENDS on that.

If I run "bitbake -D -D ..." then I see:

DEBUG: Added runtime dependencies ['libecj-bootstrap-native'] for .../meta-java/recipes-core/ecj/ecj-bootstrap-native.bb

but this dependency edge doesn't show up in task-depends.dot and, sure enough, libecj-bootstrap-native isn't getting installed into the recipe-sysroot.

Is there some other piece of logic that is squashing these dependencies? Or is there somewhere else that I ought to be looking for them?

thanks

p.

If its not in task-depends.dot, there is a much higher level problem with bitbake itself not seeing the dependencies before sstate/RSS gets anywhere near them.

I'm going to guess its some combination of -native and RDEPENDS together which are known to not work entirely "correctly".

Some example open bugs:

https://bugzilla.yoctoproject.org/show_bug.cgi?id=10113
https://bugzilla.yoctoproject.org/show_bug.cgi?id=6317

The trouble is that for native, PACKAGES = "" so how would bitbake known which RDEPENDS_<pkg> to look at? What exists today is all a bit of a hack which works for our common cases but has problems. I've been meaning to find time to take a good look at it and come up with a proper plan but so far it hasn't made the top of the agenda.

What happens if you set DEPENDS?

Cheers,

Richard

With DEPENDS I do at least get some entries for ecj-bootstrap-native in task-depends.dot. The relevant ones seem to be:

"gnujaf-native.do_prepare_recipe_sysroot" -> "ecj-bootstrap- native.do_populate_sysroot" "ecj-bootstrap-native.do_prepare_recipe_sysroot" -> "libecj-bootstrap- native.do_populate_sysroot"

which looks fairly encouraging. And in log.do_prepare_recipe_sysroot I see:

Considering setscene task: ['libecj-bootstrap-native', 'do_populate_sysroot']
  considering dependency: ['ecj-bootstrap-native', 'do_populate_sysroot']
Adding dependency on libecj-bootstrap-native

But sadly the contents of libecj-bootstrap-native doesn't actually show up in gnujaf-native's recipe-sysroot-native.

p.

By the way, this message:

                   msgbuf.append("Skipping setscene dependency %s for installation into the sysroot")

is not as illuminating as it could be. I suppose it wants "% setscenedeps[datadep][0])" on the end. :-}

p.

A bit of further sleuthing reveals that although this problem only started with rss the blame lies somewhere else. It transpires that meta-java's java-library.bbclass contains the somewhat epically unhelpful (in this context) fragment:

# Java "source" distributions often contain precompiled things

# we want to delete first.
do_removebinaries() {
  find ${WORKDIR} -name "*.jar" -exec rm {} \;
  find ${WORKDIR} -name "*.class" -exec rm {} \;
}

addtask removebinaries after do_unpack before do_patch

And, since recipe-sysroot-native is inside ${WORKDIR}, this blows away everything that rss carefully placed there. Doh.

p.

Ha, nice. I'm sure that was well intended but blasting all of ${WORKDIR} seems overkeen. I bet that was fun to chase!

Ross

good job. yeah this pre-rss assumptions, I also had another issue where clang cross compiler was happily searching native sysroot for headers. Not rss fault but exposed a latent problem.