support for multi project toolchain-cmake
A message that is not in the archive (sent off-list or not yet received): [email protected]
Hi,
Quoted 15 lines · Tobias Neumann
regarding my bug report
https://bugzilla.yoctoproject.org/show_bug.cgi?id=14703 I was forwarded to here to discuss requirements for a proper fix.In the report, I already mentioned a non-generic workaround for that. By extending the
CMAKE_FIND_ROOT_PATHwithCMAKE_INSTALL_PREFIX. However this probably doesn't cover all ways a toolchain can be used.Another solution could be to respect the user provided path and instead of overwriting the
CMAKE_FIND_ROOT_PATHvariable, to only extent it by the toolchains location.
This however would make it quite easy for a toolchain build to break isolation, when wrong user input is set. I don't know how strong your requirements regarding that would be.
Quoted 2 lines
So feedback how a fix should look like and what requirements it needs to fulfill would be appreciated.
Firstly, I'm a little unclear on your exact use case. Just to be really clear, are you using cmake within a bitbake build or are we talking about using cmake from within an SDK or eSDK external to the project?
I'm guessing this is probably an SDK question but you never mention the SDK anywhere so I wasn't entirely sure. I'll assume that in my next comment.
With the SDK, if you have two projects, one of which depends on the other, I have to wonder where the first is being installed to? Usually I'd have expected it to go into the target sysroot that the SDK toolchain is using, then it should be found by the second build? We don't support multiple sysroots for building things with the SDK.
I'm therefore still a bit puzzled about exactly what the issue is here so perhaps you could give a little more detail on the above?
Cheers,
Richard
Quoted 35 lines
Richard Purdie <[email protected]> hat am 03.02.2022 14:07 geschrieben:
Hi,
Quoted 15 lines · Tobias Neumann
regarding my bug report
https://bugzilla.yoctoproject.org/show_bug.cgi?id=14703 I was forwarded to here to discuss requirements for a proper fix.In the report, I already mentioned a non-generic workaround for that. By extending the
CMAKE_FIND_ROOT_PATHwithCMAKE_INSTALL_PREFIX. However this probably doesn't cover all ways a toolchain can be used.Another solution could be to respect the user provided path and instead of overwriting the
CMAKE_FIND_ROOT_PATHvariable, to only extent it by the toolchains location.
This however would make it quite easy for a toolchain build to break isolation, when wrong user input is set. I don't know how strong your requirements regarding that would be.Quoted 2 lines
So feedback how a fix should look like and what requirements it needs to fulfill would be appreciated.
Firstly, I'm a little unclear on your exact use case. Just to be really clear, are you using cmake within a bitbake build or are we talking about using cmake from within an SDK or eSDK external to the project?
I'm guessing this is probably an SDK question but you never mention the SDK anywhere so I wasn't entirely sure. I'll assume that in my next comment.
With the SDK, if you have two projects, one of which depends on the other, I have to wonder where the first is being installed to? Usually I'd have expected it to go into the target sysroot that the SDK toolchain is using, then it should be found by the second build? We don't support multiple sysroots for building things with the SDK.
Quoted 6 lines
I'm therefore still a bit puzzled about exactly what the issue is here so perhaps you could give a little more detail on the above?
Cheers,
Richard
Hi Richard,
thank you for your reply and please excuse that I didn't clarify my intent correctly in my first mail. Your assumption, that I meant the cmake of the SDK is correct. And your assumption regarding that the install path is different to the sysroot of the SDK toolchain is also correct.
If your statement, that multiple sysroots for building with the SDK aren't supported, is final, this discussion can be stopped here and we have to maintain extensions for that workflow in our setup. But I would like to explain our use case and why we require 2 sysroots for an SDK build.
---
We have a set of hardware platforms which are operated by a distribution generated by openembedded. This is maintained by a few developers. The hardware is used in several different projects. For the project specific development we're using the SDK with will be installed readonly below /opt. The software for these different projects can be developed without knowledge that openembedded was used underneath. For that we have tools that abstract the correct SDK selection, the environment sourcing and packaging of the final compilation. And then tools which handle the deployment on the hardware.
Because a SDK is specific for a hardware platform, but not for a project using that hardware platform, it is readonly and can be used for all projects with this hardware. This however requires that the compiled libraries generated with this SDK during the user side build process will be installed at a different location, which would be the second sysroot.
Until now we only build a cmake project with external dependencies to the installed SDK using this workflow. Further external libraries, not contained in the SDK, had to be integrated in this cmake build, either as subdirectory or via external builds (using e.g. cmake-superbuilds). We now want to extend this workflow by supporting multiple cmake projects with dependencies to each other, thus the need for this second sysroot in the build.
best regards
Tobias
Quoted 14 lines · Tobias Neumann
Because a SDK is specific for a hardware platform, but not for a project using that hardware platform, it is readonly and can be used for all projects with this hardware. This however requires that the compiled libraries generated with this SDK during the user side build process will be installed at a different location, which would be the second sysroot.
Until now we only build a cmake project with external dependencies to the installed SDK using this workflow. Further external libraries, not contained in the SDK, had to be integrated in this cmake build, either as subdirectory or via external builds (using e.g. cmake-superbuilds). We now want to extend this workflow by supporting multiple cmake projects with dependencies to each other, thus the need for this second sysroot in the build.
Quoted 2 lines
best regards
Tobias
Hi Tobias,
Any reason you set SDK to be read-only? Perhaps your use case can be covered by extensible SDK? I mean if you have some shared libraries between project, isn't that enough to extend your SDK to solve the original problem?
Cheers,
Vyacheslav
Quoted 10 lines · On Mon, 2022-02-07 at 16:30 +0100, [email protected] wrote:
thank you for your reply and please excuse that I didn't clarify my intent correctly in my first mail. Your assumption, that I meant the cmake of the SDK is correct. And your assumption regarding that the install path is different to the sysroot of the SDK toolchain is also correct.
If your statement, that multiple sysroots for building with the SDK aren't supported, is final, this discussion can be stopped here and we have to maintain extensions for that workflow in our setup. But I would like to explain our use case and why we require 2 sysroots for an SDK build.
We definitely don't support that usage scenario since it would be highly build system specific and would work in some cases but not others. I do have a suggestion of a different approach though.
Quoted 17 lines
We have a set of hardware platforms which are operated by a distribution generated by openembedded. This is maintained by a few developers. The hardware
is used in several different projects. For the project specific development we're using the SDK with will be installed readonly below /opt. The software for these different projects can be developed without knowledge that openembedded was used underneath. For that we have tools that abstract the correct SDK selection, the environment sourcing and packaging of the final compilation. And then tools which handle the deployment on the hardware.Because a SDK is specific for a hardware platform, but not for a project using that hardware platform, it is readonly and can be used for all projects with this hardware. This however requires that the compiled libraries generated with
this SDK during the user side build process will be installed at a different location, which would be the second sysroot.
Quoted 8 lines
Until now we only build a cmake project with external dependencies to the installed SDK using this workflow. Further external libraries, not contained in
the SDK, had to be integrated in this cmake build, either as subdirectory or via external builds (using e.g. cmake-superbuilds). We now want to extend this workflow by supporting multiple cmake projects with dependencies to each other,
thus the need for this second sysroot in the build.
Have you considered creating a sysroot per project with a copy of the sysroot files?
This sounds like a crazy idea at first but with hardlinks for the files it is fast and efficient to create these sysroot file trees. This is exactly what OpenEmbedded does internally and how we handle this in our builds.
Cheers,
Richard
Hello Vyacheslav and Richard,
thank you both for your feedback,
Quoted 3 lines · Vyacheslav Yurkov
Any reason you set SDK to be read-only? Perhaps your use case can be covered by extensible SDK? I mean if you have some shared libraries between project, isn't that enough to extend your SDK to solve the original problem?
While we indeed do have shared libraries between projects, they are not necessary of the same version. In general we don't want to enforce restrictions on shared code between the projects (e.g. hinder different patch sets or enforce specific versions), appart from the base distribution that is generated with OpenEmbedded. If libraries mature enough that they can be considered to be (mostly) stable, we have different processes to share it between projects.
Quoted 3 lines · Richard Purdie
We definitely don't support that usage scenario since it would be highly build system specific and would work in some cases but not others. I do have a suggestion of a different approach though.
After your first reply, I've expected this. The Bug report #14703 can then also be closed, however I can only set it to RESOLVED and not something like REJECTED, hence I would asked somebody else to reject the report.
Quoted 6 lines
Have you considered creating a sysroot per project with a copy of the sysroot files?
This sounds like a crazy idea at first but with hardlinks for the files it is fast and efficient to create these sysroot file trees. This is exactly what OpenEmbedded does internally and how we handle this in our builds.
That sounds like an interesting alternative solution. We haven't thought about it yet, but it will be something we're going to look into.
best regards
Tobias
Quoted 16 lines · On Wed, 2022-02-09 at 12:02 +0100, [email protected] wrote:
Hello Vyacheslav and Richard,
thank you both for your feedback,
Quoted 3 lines · Vyacheslav Yurkov
Any reason you set SDK to be read-only? Perhaps your use case can be covered by extensible SDK? I mean if you have some shared libraries between project, isn't that enough to extend your SDK to solve the original problem?
While we indeed do have shared libraries between projects, they are not necessary of the same version. In general we don't want to enforce restrictions on shared code between the projects (e.g. hinder different patch sets or enforce specific versions), appart from the base distribution that is generated with OpenEmbedded. If libraries mature enough that they can be considered to be (mostly) stable, we have different processes to share it between projects.
Quoted 8 lines
Quoted 3 lines · Richard Purdie
We definitely don't support that usage scenario since it would be highly build system specific and would work in some cases but not others. I do have a suggestion of a different approach though.
After your first reply, I've expected this. The Bug report #14703 can then also be closed, however I can only set it to RESOLVED and not something like REJECTED, hence I would asked somebody else to reject the report.
I've marked it as "resolved WONTFIX" and added a comment summarising the discussion here.
Quoted 10 lines
Quoted 7 lines
Have you considered creating a sysroot per project with a copy of the sysroot files?
This sounds like a crazy idea at first but with hardlinks for the files it is fast and efficient to create these sysroot file trees. This is exactly what OpenEmbedded does internally and how we handle this in our builds.
That sounds like an interesting alternative solution. We haven't thought about it yet, but it will be something we're going to look into.
I think/hope it should solve your probably and give you a relatively powerful solution that isn't cmake specific.
Cheers,
Richard