license: Ensure we find multilib packages also
RFC PATCH 1/1 license: Ensure we find multilib packages also
Make sure to find -package, this was causing a failure in the multi-lib build license generation during rootfs.
--- meta/classes/license.bbclass | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
meta/classes/license.bbclass +1 -1
@@ -88,7 +88,7 @@ license_create_manifest() { # list of installed packages is broken for deb for pkg in ${INSTALLED_PKGS}; do # not the best way to do this but licenses are not arch dependant iirc - filename=`ls ${TMPDIR}/pkgdata/*/runtime-reverse/${pkg}| head -1` + filename=`ls ${TMPDIR}/pkgdata/*/runtime-reverse/*${pkg}| head -1` pkged_pn="$(sed -n 's/^PN: //p' ${filename})" # exclude locale recipes
Signature · 1 line
1.7.7.6
Quoted 13 lines · On Thursday 13 September 2012 12:26:19 Saul Wold wrote:
Make sure to find -package, this was causing a failure in the multi-lib build license generation during rootfs.
Signed-off-by Saul Wold <[email protected]>--- meta/classes/license.bbclass | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)meta/classes/license.bbclass +0
-0
meta/classes/license.bbclass license_create_manifest() { · line 94
Quoted 8 lines
meta/classes/license.bbclass +1
-1@@ -88,7 +88,7 @@ license_create_manifest() { # list of installed packages is broken for deb for pkg in ${INSTALLED_PKGS}; do # not the best way to do this but licenses are not arch dependant iirc - filename=`ls ${TMPDIR}/pkgdata/*/runtime-reverse/${pkg}| head -1` + filename=`ls ${TMPDIR}/pkgdata/*/runtime-reverse/*${pkg}| head -1` pkged_pn="$(sed -n 's/^PN: //p' ${filename})" # exclude locale recipes
Surely this could end up matching a the wrong file when one package name is a substring of another?
Cheers,
Paul
Signature · 2 lines
Paul Eggleton Intel Open Source Technology Centre
Quoted 21 lines · Paul Eggleton
Quoted 20 lines · On Thursday 13 September 2012 12:26:19 Saul Wold wrote:
Make sure to find -package, this was causing a failure in the multi-lib build license generation during rootfs.
Signed-off-by Saul Wold <[email protected]>--- meta/classes/license.bbclass | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)meta/classes/license.bbclass +1
-1@@ -88,7 +88,7 @@ license_create_manifest() { # list of installed packages is broken for deb for pkg in ${INSTALLED_PKGS}; do # not the best way to do this but licenses are not arch dependant iirc - filename=`ls ${TMPDIR}/pkgdata/*/runtime-reverse/${pkg}| head -1` + filename=`ls ${TMPDIR}/pkgdata/*/runtime-reverse/*${pkg}| head -1` pkged_pn="$(sed -n 's/^PN: //p' ${filename})"
Quoted 5 lines
meta/classes/license.bbclass license_create_manifest() { · line 94
Quoted 1 line
meta/classes/license.bbclass +0
-0# exclude locale recipesSurely this could end up matching a the wrong file when one package name is a substring of another?
I am open to ideas here, I tried the ${MLPREFIX}, but it seems to be empty when rootfs runs, that did not help, I thought about adding the '-', but that would fail in the non-multilib case, how can I determine what the prefix will be to get a more accurate match?
Sau!
Quoted 3 lines
Cheers,
Paul
Quoted 27 lines · Saul Wold
Quoted 21 lines · Paul Eggleton
Quoted 20 lines · On Thursday 13 September 2012 12:26:19 Saul Wold wrote:
Make sure to find -package, this was causing a failure in the multi-lib build license generation during rootfs.
Signed-off-by Saul Wold <[email protected]>--- meta/classes/license.bbclass | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)meta/classes/license.bbclass +1
-1@@ -88,7 +88,7 @@ license_create_manifest() { # list of installed packages is broken for deb for pkg in ${INSTALLED_PKGS}; do # not the best way to do this but licenses are not arch dependant iirc - filename=`ls ${TMPDIR}/pkgdata/*/runtime-reverse/${pkg}| head -1` + filename=`ls ${TMPDIR}/pkgdata/*/runtime-reverse/*${pkg}| head -1` pkged_pn="$(sed -n 's/^PN: //p' ${filename})"Quoted 5 lines
meta/classes/license.bbclass license_create_manifest() { · line 94
Quoted 1 line
meta/classes/license.bbclass +0
-0# exclude locale recipesSurely this could end up matching a the wrong file when one package name is a substring of another?
Quoted 4 lines
I am open to ideas here, I tried the ${MLPREFIX}, but it seems to be empty when rootfs runs, that did not help, I thought about adding the '-', but that would fail in the non-multilib case, how can I determine what the prefix will be to get a more accurate match?
I would observe here that this probably works for ipk and multilib and only breaks for multilib+rpm. The reason is that rpm collapses the namespace when it creates the package list so "lib32-bash" becomes bash. I think this might be an error in however we generate the INSTALLED_PKGS list and we might need to revisit the rpm mechanism and ensure the multilib prefixes get added.
Cheers,
Richard
Quoted 33 lines · Richard Purdie
Quoted 27 lines · Saul Wold
Quoted 21 lines · Paul Eggleton
Quoted 20 lines · On Thursday 13 September 2012 12:26:19 Saul Wold wrote:
Make sure to find -package, this was causing a failure in the multi-lib build license generation during rootfs.
Signed-off-by Saul Wold <[email protected]>--- meta/classes/license.bbclass | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)meta/classes/license.bbclass +1
-1@@ -88,7 +88,7 @@ license_create_manifest() { # list of installed packages is broken for deb for pkg in ${INSTALLED_PKGS}; do # not the best way to do this but licenses are not arch dependant iirc - filename=`ls ${TMPDIR}/pkgdata/*/runtime-reverse/${pkg}| head -1` + filename=`ls ${TMPDIR}/pkgdata/*/runtime-reverse/*${pkg}| head -1` pkged_pn="$(sed -n 's/^PN: //p' ${filename})"Quoted 5 lines
meta/classes/license.bbclass license_create_manifest() { · line 94
Quoted 1 line
meta/classes/license.bbclass +0
-0# exclude locale recipesSurely this could end up matching a the wrong file when one package name is a substring of another?
Quoted 4 lines
I am open to ideas here, I tried the ${MLPREFIX}, but it seems to be empty when rootfs runs, that did not help, I thought about adding the '-', but that would fail in the non-multilib case, how can I determine what the prefix will be to get a more accurate match?
Quoted 6 lines
I would observe here that this probably works for ipk and multilib and only breaks for multilib+rpm. The reason is that rpm collapses the namespace when it creates the package list so "lib32-bash" becomes bash. I think this might be an error in however we generate the
INSTALLED_PKGSlist and we might need to revisit the rpm mechanism and ensure the multilib prefixes get added.
RPM has internal mechanisms to tag both the filetypes and architectures as being different, unlike most of the other packaging formats which assume a package with the same name is the same.
The way to deal with this is via the mapping actions. You should be able to map from ML_PREFIX-${PN} to ${PN}.${ARCH} and of course the reverse...
How it works....
So during package generation we know the package arch is unique and just strip off the mlprefix. To do the reverse iterate over all of the available MULTILIBS, there is code in rootfs_rpm.bbclass already to do this. Then reconstruct the package name based on the multilib configuration.
--Mark
Quoted 10 lines
Cheers,
Richard
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Quoted 39 lines · Mark Hatle
Quoted 38 lines · Richard Purdie
Quoted 36 lines · Saul Wold
Quoted 34 lines · Paul Eggleton
Quoted 25 lines · On Thursday 13 September 2012 12:26:19 Saul Wold wrote:
Make sure to find -package, this was causing a failure in the multi-lib build license generation during rootfs.
Signed-off-by Saul Wold <[email protected]>--- meta/classes/license.bbclass | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)meta/classes/license.bbclass +0
-0@@ -88,7 +88,7 @@ license_create_manifest() { # list of installed packages is broken for deb for pkg in ${INSTALLED_PKGS}; do # not the best way to do this but licenses are not archdependant iirc
- filename=`ls
${TMPDIR}/pkgdata/*/runtime-reverse/${pkg}| head -1`
+ filename=`ls
${TMPDIR}/pkgdata/*/runtime-reverse/*${pkg}| head -1`
pkged_pn="$(sed -n 's/^PN: //p' ${filename})"Quoted 1 line
# exclude locale recipes
Surely this could end up matching a the wrong file when one package name is a
substring of another?
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
I am open to ideas here, I tried the ${MLPREFIX}, but it seems to be empty when rootfs runs, that did not help, I thought about adding the '-', but that would fail in the non-multilib case, how can I determine what the prefix will be to get a more accurate match?
I'm not particularly happy with how we've been doing manifests for a while now and have been hacking at a revamp of it for the past 2 months on and off.
We really shouldn't be relying on package data at all for license manifests. As the manifests are relying on list_installed_packages, they end up inaccurate anyway as list_installed_packages does exactly that. List software installed via package. Not installed via the package manager? Then you aren't in the manifest. This ends up missing quite a few things necessary from a release engineering standpoint (like, hey, modutils? The kernel? Not listed in the manifest. IMHO this is wrong.)
As a release engineer, I kind of don't really care if the software on my image was installed via the packaging system. I just want a list of what is on the images so I can audit my images to ensure GPL commitments.
One way I've been looking at fixing this is via event handlers. I'll have some code for RFC on this hopefully soon, but a preview (which may or may not work it's still a WIP) is at:
http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=eflanagan/license_manifest
One thing the patch I've been working on does other than creates accurate manifests is it also tries to create a semi-compliant SPDX 1.0 manifest.
I know this is late in the cycle so it might not make 1.3, but this may be something we want to look at to end up solving this issue once and for all.
-b
Quoted 44 lines
Quoted 8 lines
I would observe here that this probably works for ipk and multilib and only breaks for multilib+rpm. The reason is that rpm collapses the namespace when it creates the package list so "lib32-bash" becomes bash. I think this might be an error in however we generate the
INSTALLED_PKGSlist and we might need to revisit the rpm mechanism and ensure the multilib prefixes get added.RPM has internal mechanisms to tag both the filetypes and architectures as being different, unlike most of the other packaging formats which assume a package with the same name is the same.
The way to deal with this is via the mapping actions. You should be able to map from ML_PREFIX-${PN} to ${PN}.${ARCH} and of course the reverse...
How it works....
So during package generation we know the package arch is unique and just strip off the mlprefix. To do the reverse iterate over all of the available MULTILIBS, there is code in rootfs_rpm.bbclass already to do this. Then reconstruct the package name based on the multilib configuration.
--Mark
Quoted 10 lines
Cheers,
Richard
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Signature · 3 lines
Elizabeth Flanagan Yocto Project Build and Release
Quoted 69 lines · Elizabeth
Quoted 39 lines · Mark Hatle
Quoted 38 lines · Richard Purdie
Quoted 36 lines · Saul Wold
Quoted 34 lines · Paul Eggleton
Quoted 25 lines · On Thursday 13 September 2012 12:26:19 Saul Wold wrote:
Make sure to find -package, this was causing a failure in the multi-lib build license generation during rootfs.
Signed-off-by Saul Wold <[email protected]>--- meta/classes/license.bbclass | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)meta/classes/license.bbclass +0
-0@@ -88,7 +88,7 @@ license_create_manifest() { # list of installed packages is broken for deb for pkg in ${INSTALLED_PKGS}; do # not the best way to do this but licenses are not archdependant iirc
- filename=`ls
${TMPDIR}/pkgdata/*/runtime-reverse/${pkg}| head -1`
+ filename=`ls
${TMPDIR}/pkgdata/*/runtime-reverse/*${pkg}| head -1`
pkged_pn="$(sed -n 's/^PN: //p' ${filename})"Quoted 1 line
# exclude locale recipes
Surely this could end up matching a the wrong file when one package name is a
substring of another?Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
I am open to ideas here, I tried the ${MLPREFIX}, but it seems to be empty when rootfs runs, that did not help, I thought about adding the '-', but that would fail in the non-multilib case, how can I determine what the prefix will be to get a more accurate match?
I'm not particularly happy with how we've been doing manifests for a while now and have been hacking at a revamp of it for the past 2 months on and off.
We really shouldn't be relying on package data at all for license manifests. As the manifests are relying on
list_installed_packages, they end up inaccurate anyway aslist_installed_packagesdoes exactly that. List software installed via package. Not installed via the package manager? Then you aren't in the manifest. This ends up missing quite a few things necessary from a release engineering standpoint (like, hey, modutils? The kernel? Not listed in the manifest. IMHO this is wrong.)As a release engineer, I kind of don't really care if the software on my image was installed via the packaging system. I just want a list of what is on the images so I can audit my images to ensure GPL commitments.
One way I've been looking at fixing this is via event handlers. I'll have some code for RFC on this hopefully soon, but a preview (which may or may not work it's still a WIP) is at:
http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=eflanagan/license_manifest
Quoted 7 lines
One thing the patch I've been working on does other than creates accurate manifests is it also tries to create a semi-compliant SPDX 1.0 manifest.
I know this is late in the cycle so it might not make 1.3, but this may be something we want to look at to end up solving this issue once and for all.
This depends whether this is a release manifest or an image manifest. You likely want *both* under different circumstances. If the kernel is installed in the image, it will be listed in the image manifest. We just need to be really clear about what the manifest is for.
As for modutils, we use kmod now. Basically anything doing deployment should have a manifest associated with it. That is do_deploy (kernel, uboot, some small list of others) and do_rootfs. do_package_write_* are also arguably deployment in another form, as are sstate feeds.
Cheers,
Richard
Quoted 8 lines · Elizabeth
We really shouldn't be relying on package data at all for license manifests. As the manifests are relying on
list_installed_packages, they end up inaccurate anyway aslist_installed_packagesdoes exactly that. List software installed via package. Not installed via the package manager? Then you aren't in the manifest. This ends up missing quite a few things necessary from a release engineering standpoint (like, hey, modutils? The kernel? Not listed in the manifest. IMHO this is wrong.)
What do you mean by modutils? If you mean the earlier suggestion that module- init-tools was missing from the manifest, I'm fairly certain that was a misunderstanding because of it being replaced by kmod.
The thing is, we have no other means of accurately determining the contents of the image than the installed package list, given that the package manager is ultimately in charge of resolving and installing dependencies during image creation. The list may not cover additional files copied into tmp/deploy as part of building the image (kernel, bootloader, etc.) - however, that does not make it inaccurate as to the contents of the image, let's be clear about that.
I agree we do need to write out the license information for these additional files; however, since they aren't actually *in* the image itself, I think we need to list these in a separate file. What happens for example if the system builds u-boot as a result of building my image, but I don't ever end up distributing that with the image?
As to the mechanism for picking these up, the only real possibility I can see is to hook into do_deploy; this is not currently straightforward though since that's not something that is implemented generically at the moment.
Cheers,
Paul
Signature · 2 lines
Paul Eggleton Intel Open Source Technology Centre
Quoted 10 lines · Paul Eggleton
Quoted 8 lines · Elizabeth
We really shouldn't be relying on package data at all for license manifests. As the manifests are relying on
list_installed_packages, they end up inaccurate anyway aslist_installed_packagesdoes exactly that. List software installed via package. Not installed via the package manager? Then you aren't in the manifest. This ends up missing quite a few things necessary from a release engineering standpoint (like, hey, modutils? The kernel? Not listed in the manifest. IMHO this is wrong.)
Quoted 4 lines
What do you mean by modutils? If you mean the earlier suggestion that module- init-tools was missing from the manifest, I'm fairly certain that was a misunderstanding because of it being replaced by kmod.
Correct. Even so, there are things obviously missing from the manifests. Kernel is the most obvious.
Quoted 7 lines
The thing is, we have no other means of accurately determining the contents of the image than the installed package list, given that the package manager is ultimately in charge of resolving and installing dependencies during image creation. The list may not cover additional files copied into tmp/deploy as part of building the image (kernel, bootloader, etc.) - however, that does not make it inaccurate as to the contents of the image, let's be clear about that.
No, but it's like the joke about the two software managers in a hot air balloon asking the engineer on the group where they are only to be told "In a hot air balloon". It's technically correct but the user case I keep running into here is that people want to know what they're deploying with a product. The expectation people have (and right or wrong, it's what I keep running into) is that everything that's on the hddimg is what is on the manifest.
Quoted 5 lines
I agree we do need to write out the license information for these additional files; however, since they aren't actually *in* the image itself, I think we need to list these in a separate file. What happens for example if the system builds u-boot as a result of building my image, but I don't ever end up distributing that with the image?
Then you have a manifest that has software that you end up complying with the GPL (when you don't really need to) verses having a manifest that doesn't list software which you do need to comply with the GPL. Both are wrong, but one is more wrong.
RP is right here. We probably need to list a few different use cases here and support the most common ones.
Quoted 4 lines
As to the mechanism for picking these up, the only real possibility I can see is to hook into
do_deploy; this is not currently straightforward though since that's not something that is implemented generically at the moment.
I'm currently hooking into do_package because, as best I can figure it out, everything ends up getting packaged, even if it's not installed via package. I'm happy for other suggestions though.
-b
Quoted 8 lines
Cheers,
Paul--
Paul Eggleton
Intel Open Source Technology Centre
Signature · 3 lines
Elizabeth Flanagan Yocto Project Build and Release
Quoted 28 lines · Elizabeth
Quoted 10 lines · Paul Eggleton
Quoted 8 lines · Elizabeth
We really shouldn't be relying on package data at all for license manifests. As the manifests are relying on
list_installed_packages, they end up inaccurate anyway aslist_installed_packagesdoes exactly that. List software installed via package. Not installed via the package manager? Then you aren't in the manifest. This ends up missing quite a few things necessary from a release engineering standpoint (like, hey, modutils? The kernel? Not listed in the manifest. IMHO this is wrong.)Quoted 4 lines
What do you mean by modutils? If you mean the earlier suggestion that module- init-tools was missing from the manifest, I'm fairly certain that was a misunderstanding because of it being replaced by kmod.
Correct. Even so, there are things obviously missing from the manifests. Kernel is the most obvious.
Quoted 7 lines
The thing is, we have no other means of accurately determining the contents of the image than the installed package list, given that the package manager is ultimately in charge of resolving and installing dependencies during image creation. The list may not cover additional files copied into tmp/deploy as part of building the image (kernel, bootloader, etc.) - however, that does not make it inaccurate as to the contents of the image, let's be clear about that.
Quoted 7 lines
No, but it's like the joke about the two software managers in a hot air balloon asking the engineer on the group where they are only to be told "In a hot air balloon". It's technically correct but the user case I keep running into here is that people want to know what they're deploying with a product. The expectation people have (and right or wrong, it's what I keep running into) is that everything that's on the hddimg is what is on the manifest.
From thinking about the deployment of the rootfs through images, it seems to me it should be the responsibility of the components populating the items to produce a manifest of what they've done. Be it install a package, or copy a file. Then at the end collect together these manifest fragments to describe exactly what was done (and where it was done to!).
Quoted 36 lines
Quoted 5 lines
I agree we do need to write out the license information for these additional files; however, since they aren't actually *in* the image itself, I think we need to list these in a separate file. What happens for example if the system builds u-boot as a result of building my image, but I don't ever end up distributing that with the image?
Then you have a manifest that has software that you end up complying with the GPL (when you don't really need to) verses having a manifest that doesn't list software which you do need to comply with the GPL. Both are wrong, but one is more wrong.
RP is right here. We probably need to list a few different use cases here and support the most common ones.
Quoted 4 lines
As to the mechanism for picking these up, the only real possibility I can see is to hook into
do_deploy; this is not currently straightforward though since that's not something that is implemented generically at the moment.I'm currently hooking into
do_packagebecause, as best I can figure it out, everything ends up getting packaged, even if it's not installed via package. I'm happy for other suggestions though.-b
Quoted 8 lines
Cheers,
Paul--
Paul Eggleton
Intel Open Source Technology Centre
Quoted 8 lines · Elizabeth
Quoted 7 lines
The thing is, we have no other means of accurately determining the contents of the image than the installed package list, given that the package manager is ultimately in charge of resolving and installing dependencies during image creation. The list may not cover additional files copied into tmp/deploy as part of building the image (kernel, bootloader, etc.) - however, that does not make it inaccurate as to the contents of the image, let's be clear about that.
Quoted 5 lines
No, but it's like the joke about the two software managers in a hot air balloon asking the engineer on the group where they are only to be told "In a hot air balloon". It's technically correct but the user case I keep running into here is that people want to know what they're deploying with a product.
Again, I absolutely agree we need to report the licenses for these files. I'm just advocating not commingling two separate things.
Quoted 3 lines
The expectation people have (and right or
wrong, it's what I keep running into) is that everything that's on the hddimg is what is on the manifest.
hddimg files are a special situation anyway though. To handle those properly we would have to use some special license reporting in the same place where the hddimg is composed, because we cannot really tell for certain anywhere else in the code as to what is in it.
Quoted 6 lines
Quoted 5 lines
I agree we do need to write out the license information for these additional files; however, since they aren't actually *in* the image itself, I think we need to list these in a separate file. What happens for example if the system builds u-boot as a result of building my image, but I don't ever end up distributing that with the image?
Quoted 7 lines
Then you have a manifest that has software that you end up complying with the GPL (when you don't really need to) verses having a manifest that doesn't list software which you do need to comply with the GPL. Both are wrong, but one is more wrong.
RP is right here. We probably need to list a few different use cases here and support the most common ones.
I think we can satisfy all of these cases in the manner I described - a manifest for the image (what we already have) and an additional manifest beside it that lists all of the additionally deployed files along with their license information. If the files are right next to eachother with the same base name there can't be any confusion.
Quoted 8 lines
Quoted 4 lines
As to the mechanism for picking these up, the only real possibility I can see is to hook into
do_deploy; this is not currently straightforward though since that's not something that is implemented generically at the moment.I'm currently hooking into
do_packagebecause, as best I can figure it out, everything ends up getting packaged, even if it's not installed via package. I'm happy for other suggestions though.
pkgdata (which writes files into TMPDIR/pkgdata) already hooks into do_package and already lists LICENSE on a per-package basis for every package written out. Clearly we can't just take absolutely every package produced as the list for the manifest though, because not all of that gets installed into the image. Another more serious issue is that it's not guaranteed that everything that gets deployed is packaged, so this probably wouldn't satisfactorily solve the issue anyway.
As far as I can see, do_deploy is the only way, since we want to know what is deployed and which recipe deployed it. We'll just have to adjust things so that there is a central point we can tap into for do_deploy in order to collect the desired information - I haven't looked into how yet but I don't anticipate it being too hard.
I think we're also going to have to add a specific call that can be made from anywhere during do_rootfs that adds a recipe to be reported in the manifest. We can use this for custom cases such as hddimg and provide it for anyone doing anything similar to call manually (e.g. adding non-packaged files into their image in a post-processing function - not really recommended but I know people do do it).
Cheers,
Paul
Signature · 2 lines
Paul Eggleton Intel Open Source Technology Centre