About PACKAGECONFIG audit
Hi All,
We have seen some problems like this:
$ bitbake libcap
ld: warning: libaudit.so.1, needed by .../lib64/libpam.so, not found .../lib64/libpam.so: undefined reference to audit_open' .../lib64/libpam.so: undefined reference to audit_close'
......
The libaudit has been available and libpam build with it, after we run another build with sstate-cache and w/o libaudit, the error occurs. The solution in these cases is to use PACKAGECONFIG to lay out dependencies on optional packages and make them explicit. We need to run an audit and catch all of these issues.
A probable ways maybe:
1) Run world build to generate the sysroots as much as possible;
2) Run 'configure --help' on each package to spot all of the optional dependencies, and convert them into PACKAGECONFIG specs/or any other prompts.
3) Run #1, #2 as an audit script periodically.
Or could we run this check as part of a QA build step?
Thanks
Wenzong
Quoted 7 lines · wenzong fan
A probable ways maybe:
1) Run world build to generate the sysroots as much as possible;
2) Run 'configure --help' on each package to spot all of the optional dependencies, and convert them into PACKAGECONFIG specs/or any other prompts.
3) Run #1, #2 as an audit script periodically.
Step 2 would be hard to to automate and some optional dependencies don't have configure options. However Martin Jansa recently contributed scripts/test-dependencies.sh which does repeated builds to determine if there are any "floating" or missing dependencies, which can be ran periodically as an audit.
Ross
Quoted 1 line · wenzong fan
Or could we run this check as part of a QA build step?
I think that could be done, in principle. Now that "ld --no-copy-dt-needed-entries" is the default, there's no reason that the shlibs code in package.bbclass couldn't be taught to verify that all the shared library providers it identifies come from recipes that are listed in DEPENDS.
The only difficulty is that you'd need to accept DEPENDS-of-DEPENDS (e.g. a recipe which DEPENDS on gtk+ but also links with libatk or something) and I don't think there's currently any straightforward way for package.bbclass to generate the recursive dependency chain for an arbitrary recipe. An easy if inelegant workaround would be for it to just stash the value of ${DEPENDS} along with the other shlibs data that it writes out for each provider.
p.
Hi Wenzong,
Quoted 15 lines · On Thursday 15 August 2013 17:29:46 wenzong fan wrote:
We have seen some problems like this:
$ bitbake libcapld: warning: libaudit.so.1, needed by .../lib64/libpam.so, not found .../lib64/libpam.so: undefined reference to
audit_open' .../lib64/libpam.so: undefined reference toaudit_close'
......The libaudit has been available and libpam build with it, after we run another build with sstate-cache and w/o libaudit, the error occurs. The solution in these cases is to use PACKAGECONFIG to lay out dependencies on optional packages and make them explicit. We need to run an audit and catch all of these issues.
Quoted 3 lines
A probable ways maybe:
1) Run world build to generate the sysroots as much as possible;
(See Ross's reply for a pointer to the new test-dependencies script)
Quoted 3 lines
2) Run 'configure --help' on each package to spot all of the optional dependencies, and convert them into PACKAGECONFIG specs/or any other prompts.
Actually I have started along this line as well, putting the output of configure --help into buildhistory so you can see the differences. I'm not totally sure how useful it is at the moment and it will need some smooting out to avoid known differences (paths etc.).
Cheers,
Paul
Signature · 2 lines
Paul Eggleton Intel Open Source Technology Centre
Quoted 18 lines · Paul Eggleton
Hi Wenzong,
Quoted 15 lines · On Thursday 15 August 2013 17:29:46 wenzong fan wrote:
We have seen some problems like this:
$ bitbake libcapld: warning: libaudit.so.1, needed by .../lib64/libpam.so, not found .../lib64/libpam.so: undefined reference to
audit_open' .../lib64/libpam.so: undefined reference toaudit_close'
......The libaudit has been available and libpam build with it, after we run another build with sstate-cache and w/o libaudit, the error occurs. The solution in these cases is to use PACKAGECONFIG to lay out dependencies on optional packages and make them explicit. We need to run an audit and catch all of these issues.
Quoted 5 lines
Quoted 3 lines
A probable ways maybe:
1) Run world build to generate the sysroots as much as possible;
(See Ross's reply for a pointer to the new test-dependencies script)
That would be a good start point, I have run that script once, but looks not any failure occurs.
I'll find a box and run it periodically.
Thanks
Wenzong
Quoted 13 lines
Quoted 4 lines
2) Run 'configure --help' on each package to spot all of the optional dependencies, and convert them into PACKAGECONFIG specs/or any other prompts.
Actually I have started along this line as well, putting the output of configure --help into buildhistory so you can see the differences. I'm not totally sure how useful it is at the moment and it will need some smooting out to avoid known differences (paths etc.).
Cheers,
Paul
Quoted 9 lines · Phil Blundell
Quoted 1 line · wenzong fan
Or could we run this check as part of a QA build step?
I think that could be done, in principle. Now that "ld --no-copy-dt-needed-entries" is the default, there's no reason that the shlibs code in package.bbclass couldn't be taught to verify that all the shared library providers it identifies come from recipes that are listed in DEPENDS.
Quoted 7 lines
The only difficulty is that you'd need to accept DEPENDS-of-DEPENDS (e.g. a recipe which DEPENDS on gtk+ but also links with libatk or something) and I don't think there's currently any straightforward way for package.bbclass to generate the recursive dependency chain for an arbitrary recipe. An easy if inelegant workaround would be for it to just stash the value of ${DEPENDS} along with the other shlibs data that it writes out for each provider.
Something like that is outlined in WIP patch for
https://bugzilla.yoctoproject.org/show_bug.cgi?id=4628
but I got stuck on generating this recursive dependency chain
Signature · 1 line
Martin 'JaMa' Jansa jabber: [email protected]
Quoted 19 lines · wenzong fan
Quoted 18 lines · Paul Eggleton
Hi Wenzong,
Quoted 15 lines · On Thursday 15 August 2013 17:29:46 wenzong fan wrote:
We have seen some problems like this:
$ bitbake libcapld: warning: libaudit.so.1, needed by .../lib64/libpam.so, not found .../lib64/libpam.so: undefined reference to
audit_open' .../lib64/libpam.so: undefined reference toaudit_close'
......The libaudit has been available and libpam build with it, after we run another build with sstate-cache and w/o libaudit, the error occurs. The solution in these cases is to use PACKAGECONFIG to lay out dependencies on optional packages and make them explicit. We need to run an audit and catch all of these issues.
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
A probable ways maybe:
1) Run world build to generate the sysroots as much as possible;
(See Ross's reply for a pointer to the new test-dependencies script)
That would be a good start point, I have run that script once, but looks not any failure occurs.
I don't believe it, see "State of bitbake world" thread where I've sent results from last 3 executions.
There is plenty of issues for everybody :)
Signature · 1 line
Martin 'JaMa' Jansa jabber: [email protected]
Quoted 9 lines · Phil Blundell
Quoted 1 line · wenzong fan
Or could we run this check as part of a QA build step?
I think that could be done, in principle. Now that "ld --no-copy-dt-needed-entries" is the default, there's no reason that the shlibs code in package.bbclass couldn't be taught to verify that all the shared library providers it identifies come from recipes that are listed in DEPENDS.
Quoted 7 lines
The only difficulty is that you'd need to accept DEPENDS-of-DEPENDS (e.g. a recipe which DEPENDS on gtk+ but also links with libatk or something) and I don't think there's currently any straightforward way for package.bbclass to generate the recursive dependency chain for an arbitrary recipe. An easy if inelegant workaround would be for it to just stash the value of ${DEPENDS} along with the other shlibs data that it writes out for each provider.
Bitbake itself has that information and I suspect we should share it to tasks. The exact form of that would be the harder thing to determine...
Cheers,
Richard
Quoted 7 lines · Richard Purdie
Quoted 3 lines · Phil Blundell
I don't think there's currently any straightforward way for package.bbclass to generate the recursive dependency chain for an arbitrary recipe.
Bitbake itself has that information and I suspect we should share it to tasks. The exact form of that would be the harder thing to determine...
If it's something that bitbake already computes (or could compute cheaply) then you could just arrange for it to add it to the datastore as something like ${BB_COMPUTED_DEPENDS} before running each task. Just a flat textual list of recipes, de-duped but in arbitrary order, would probably be good enough for all the use-cases we've talked about so far. I guess the question is whether you need to do anything clever with selecting the dependencies for particular tasks, or whether you could just take the dependencies for all tasks that the recipe defines and collapse them into a single list. I think probably the latter.
If it's expensive to compute then I guess there'd need to be some sort of bb.cooker.get_recursive_dependencies() kind of function that tasks can call when they want the information.
p.
Add Joe Slater who owns US17463: PACKAGECONFIG clean-up.
Quoted 17 lines · wenzong fan
Hi All,
We have seen some problems like this:
$ bitbake libcapld: warning: libaudit.so.1, needed by .../lib64/libpam.so, not found .../lib64/libpam.so: undefined reference to
audit_open' .../lib64/libpam.so: undefined reference toaudit_close'
......The libaudit has been available and libpam build with it, after we run another build with sstate-cache and w/o libaudit, the error occurs. The solution in these cases is to use PACKAGECONFIG to lay out dependencies on optional packages and make them explicit. We need to run an audit and catch all of these issues.
Quoted 6 lines
A probable ways maybe:
1) Run world build to generate the sysroots as much as possible;
2) Run 'configure --help' on each package to spot all of the optional dependencies, and convert them into PACKAGECONFIG specs/or any other prompts.
We should try to automate or codify this step so that the output is a list of oe-core package names.
Quoted 3 lines
3) Run #1, #2 as an audit script periodically.
Or could we run this check as part of a QA build step?
Do you mean:
scripts/test-dependencies.sh
// Randy
Quoted 4 lines
Thanks
Wenzong
Signature · 2 lines
# Randy MacLeod. SMTS, Linux, Wind River Direct: 613.963.1350
Quoted 31 lines · Randy MacLeod
Add Joe Slater who owns US17463: PACKAGECONFIG clean-up.
Quoted 17 lines · wenzong fan
Hi All,
We have seen some problems like this:
$ bitbake libcapld: warning: libaudit.so.1, needed by .../lib64/libpam.so, not found .../lib64/libpam.so: undefined reference to
audit_open' .../lib64/libpam.so: undefined reference toaudit_close'
......The libaudit has been available and libpam build with it, after we run another build with sstate-cache and w/o libaudit, the error occurs. The solution in these cases is to use PACKAGECONFIG to lay out dependencies on optional packages and make them explicit. We need to run an audit and catch all of these issues.
Quoted 6 lines
A probable ways maybe:
1) Run world build to generate the sysroots as much as possible;
2) Run 'configure --help' on each package to spot all of the optional dependencies, and convert them into PACKAGECONFIG specs/or any other prompts.We should try to automate or codify this step so that the output is a list of oe-core package names.
Quoted 6 lines
Quoted 3 lines
3) Run #1, #2 as an audit script periodically.
Or could we run this check as part of a QA build step?
Do you mean:
scripts/test-dependencies.sh
No, I mean some check steps that if PACKAGECONFIG should be used, not limit to the script.
Thanks
Wenzong
Quoted 9 lines
// Randy
Quoted 4 lines
Thanks
Wenzong
Quoted 34 lines · wenzong fan
Quoted 33 lines · Randy MacLeod
Add Joe Slater who owns US17463: PACKAGECONFIG clean-up.
Quoted 17 lines · wenzong fan
Hi All,
We have seen some problems like this:
$ bitbake libcapld: warning: libaudit.so.1, needed by .../lib64/libpam.so, not found .../lib64/libpam.so: undefined reference to
audit_open' .../lib64/libpam.so: undefined reference toaudit_close'
......The libaudit has been available and libpam build with it, after we run another build with sstate-cache and w/o libaudit, the error occurs. The solution in these cases is to use PACKAGECONFIG to lay out dependencies on optional packages and make them explicit. We need to run an audit and catch all of these issues.
Quoted 6 lines
A probable ways maybe:
1) Run world build to generate the sysroots as much as possible;
2) Run 'configure --help' on each package to spot all of the optional dependencies, and convert them into PACKAGECONFIG specs/or any other prompts.We should try to automate or codify this step so that the output is a list of oe-core package names.
Quoted 2 lines
3) Run #1, #2 as an audit script periodically.
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
Or could we run this check as part of a QA build step?
Do you mean:
scripts/test-dependencies.shNo, I mean some check steps that if PACKAGECONFIG should be used, not limit to the script.
thread replace by discussion on:
[Bug 5033] New: Way to audit all optional dependencies
https://bugzilla.yoctoproject.org/show_bug.cgi?id=5033
// Randy
Quoted 13 lines
Thanks
WenzongQuoted 9 lines
// Randy
Quoted 4 lines
Thanks
Wenzong
Signature · 2 lines
# Randy MacLeod. SMTS, Linux, Wind River Direct: 613.963.1350