Fix incremental rpm image generation
cover Fix incremental rpm image generation
The following changes since commit 043871d7e5d2d19c2ff43e54d2ff180c09e8903e:
kern-tools: integrate minor fixes (2012-05-08 16:06:15 +0100)
are available in the git repository at:
git://git.pokylinux.org/poky-contrib robert/rpm_inc
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=robert/rpm_inc
Robert Yang (2):
package_rpm.bbclass: Fix incremental rpm image generation rootfs_rpm.bbclass: don't remove rpmlib when INC_RPM_IMAGE_GEN=1
meta/classes/package_rpm.bbclass | 20 ++++++++------------ meta/classes/rootfs_rpm.bbclass | 7 +++++-- 2 files changed, 13 insertions(+), 14 deletions(-)
PATCH 1/2 package_rpm.bbclass: Fix incremental rpm image generation
Fix the incremental rpm image generation, it didn't work since the code has been changed.
The btmanifest should have a ".manifest" suffix, so that it can be moved to ${T} by rootfs_rpm.bbclass:
mv ${IMAGE_ROOTFS}/install/*.manifest ${T}/
Note: The locale pkgs would always be re-installed.
[YOCTO #2440]
--- meta/classes/package_rpm.bbclass | 20 ++++++++------------ 1 files changed, 8 insertions(+), 12 deletions(-)
meta/classes/package_rpm.bbclass +8 -12
@@ -158,28 +158,24 @@ rpm_common_comand () { rpm_update_pkg () { manifest=$1 - btmanifest=$manifest.bt + btmanifest=$manifest.bt.manifest + pre_btmanifest=${T}/${btmanifest##/*/} local target_rootfs="${INSTALL_ROOTFS_RPM}" # Save the rpm's build time for incremental image generation, and the file # would be moved to ${T} - rm -f $btmanifest for i in `cat $manifest`; do # Use "rpm" rather than "${RPM}" here, since we don't need the # '--dbpath' option - echo "$i `rpm -qp --qf '%{BUILDTIME}\n' $i`" >> $btmanifest - done + echo "$i `rpm -qp --qf '%{BUILDTIME}\n' $i`" + done | sort -u > $btmanifest # Only install the different pkgs if incremental image generation is set - if [ "${INC_RPM_IMAGE_GEN}" = "1" -a -f ${T}/total_solution_bt.manifest -a \ + if [ "${INC_RPM_IMAGE_GEN}" = "1" -a -f "$pre_btmanifest" -a \ "${IMAGE_PKGTYPE}" = "rpm" ]; then - cur_list="$btmanifest" - pre_list="${T}/total_solution_bt.manifest" - sort -u $cur_list -o $cur_list - sort -u $pre_list -o $pre_list - comm -1 -3 $cur_list $pre_list | sed 's#.*/\(.*\)\.rpm .*#\1#' > \ + comm -1 -3 $btmanifest $pre_btmanifest | sed 's#.*/\(.*\)\.rpm .*#\1#' > \ ${target_rootfs}/install/remove.manifest - comm -2 -3 $cur_list $pre_list | awk '{print $1}' > \ + comm -2 -3 $btmanifest $pre_btmanifest | awk '{print $1}' > \ ${target_rootfs}/install/incremental.manifest # Attempt to remove unwanted pkgs, the scripts(pre, post, etc.) has not
@@ -462,7 +458,7 @@ EOF # probably a feature. The only way to convince rpm to actually run the preinstall scripts # for base-passwd and shadow first before installing packages that depend on these packages # is to do two image installs, installing one set of packages, then the other. - if [ "${INC_RPM_IMAGE_GEN}" = "1" -a -f ${T}/total_solution_bt.manifest ]; then + if [ "${INC_RPM_IMAGE_GEN}" = "1" -a -f "$pre_btmanifest" ]; then echo "Skipping pre install due to exisitng image" else echo "# Initial Install manifest" > ${target_rootfs}/install/initial_install.manifest
Signature · 1 line
1.7.1
PATCH 2/2 rootfs_rpm.bbclass: don't remove rpmlib when INC_RPM_IMAGE_GEN=1
In reply to Robert YangDon't remove rpmlib when INC_RPM_IMAGE_GEN=1, otherwise the list_installed_packages would get nothing in the second build.
[YOCTO #2440]
--- meta/classes/rootfs_rpm.bbclass | 7 +++++-- 1 files changed, 5 insertions(+), 2 deletions(-)
meta/classes/rootfs_rpm.bbclass +5 -2
@@ -154,8 +154,11 @@ EOF } remove_packaging_data_files() { - rm -rf ${IMAGE_ROOTFS}${rpmlibdir} - rm -rf ${IMAGE_ROOTFS}${opkglibdir} + # Don't remove the rpmlib when INC_RPM_IMAGE_GEN=1 + if [ "${INC_RPM_IMAGE_GEN}" != "1" ]; then + rm -rf ${IMAGE_ROOTFS}${rpmlibdir} + rm -rf ${IMAGE_ROOTFS}${opkglibdir} + fi } RPM_QUERY_CMD = '${RPM} --root ${IMAGE_ROOTFS} -D "_dbpath ${rpmlibdir}" \
Signature · 1 line
1.7.1
Re: [PATCH 2/2] rootfs_rpm.bbclass: don't remove rpmlib when INC_RPM_IMAGE_GEN=1
Op 9 mei 2012, om 10:20 heeft Robert Yang het volgende geschreven:
Quoted 2 lines
Don't remove rpmlib when INC_RPM_IMAGE_GEN=1, otherwise the
list_installed_packageswould get nothing in the second build.
What happens when I do a build, enable INC_RPM_IMAGE_GEN and do another build?
Re: [PATCH 2/2] rootfs_rpm.bbclass: don't remove rpmlib when INC_RPM_IMAGE_GEN=1
Quoted 7 lines · Koen Kooi
Op 9 mei 2012, om 10:20 heeft Robert Yang het volgende geschreven:
Quoted 2 lines
Don't remove rpmlib when INC_RPM_IMAGE_GEN=1, otherwise the
list_installed_packageswould get nothing in the second build.What happens when I do a build, enable
INC_RPM_IMAGE_GENand do another build?
The intention is that only the packages that have been changed/upgraded will be installed. Instead of the whole image being generated from scratch each time.
--Mark
Re: [PATCH 2/2] rootfs_rpm.bbclass: don't remove rpmlib when INC_RPM_IMAGE_GEN=1
Op 9 mei 2012, om 16:52 heeft Mark Hatle het volgende geschreven:
Quoted 10 lines · Koen Kooi
Quoted 7 lines
Op 9 mei 2012, om 10:20 heeft Robert Yang het volgende geschreven:
Quoted 2 lines
Don't remove rpmlib when INC_RPM_IMAGE_GEN=1, otherwise the
list_installed_packageswould get nothing in the second build.What happens when I do a build, enable
INC_RPM_IMAGE_GENand do another build?The intention is that only the packages that have been changed/upgraded will be installed. Instead of the whole image being generated from scratch each time.
That's what INC_RPM_IMAGE_GEN does, yes. What I'm asking is how this patch addresses the sequence I outlined above.
Re: [PATCH 2/2] rootfs_rpm.bbclass: don't remove rpmlib when INC_RPM_IMAGE_GEN=1
Quoted 7 lines · Koen Kooi
Op 9 mei 2012, om 16:52 heeft Mark Hatle het volgende geschreven:
Quoted 4 lines · Koen Kooi
Quoted 3 lines
Op 9 mei 2012, om 10:20 heeft Robert Yang het volgende geschreven:
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
Don't remove rpmlib when INC_RPM_IMAGE_GEN=1, otherwise the
list_installed_packageswould get nothing in the second build.What happens when I do a build, enable
INC_RPM_IMAGE_GENand do another build?The intention is that only the packages that have been changed/upgraded will be installed. Instead of the whole image being generated from scratch each time.
That's what
INC_RPM_IMAGE_GENdoes, yes. What I'm asking is how this patch addresses the sequence I outlined above.
Explain what you mean by "build".
Using the following:
. ./oe-init-build-env build-test
<setup local.conf w/ rpm packaging and INC_RPM_IMAGE_GEN> bitbake core-image-core
--- time passes, git pull pulls in changes --- bitbake core-image-core
The second "build", only the changed packages will be added to the image.
You you are using different build directories, then it doesn't re-use anything.
The purpose of this is incremental software development within a single build
directory.
--Mark
Quoted 4 lines
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Re: [PATCH 2/2] rootfs_rpm.bbclass: don't remove rpmlib when INC_RPM_IMAGE_GEN=1
Op 9 mei 2012, om 17:15 heeft Mark Hatle het volgende geschreven:
Quoted 27 lines · Koen Kooi
Quoted 7 lines
Op 9 mei 2012, om 16:52 heeft Mark Hatle het volgende geschreven:
Quoted 4 lines · Koen Kooi
Quoted 3 lines
Op 9 mei 2012, om 10:20 heeft Robert Yang het volgende geschreven:
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
Don't remove rpmlib when INC_RPM_IMAGE_GEN=1, otherwise the
list_installed_packageswould get nothing in the second build.What happens when I do a build, enable
INC_RPM_IMAGE_GENand do another build?The intention is that only the packages that have been changed/upgraded will be installed. Instead of the whole image being generated from scratch each time.
That's what
INC_RPM_IMAGE_GENdoes, yes. What I'm asking is how this patch addresses the sequence I outlined above.Explain what you mean by "build".
Using the following:
. ./oe-init-build-env build-test
<setup local.conf w/ rpm packaging andINC_RPM_IMAGE_GEN> bitbake core-image-core
--- time passes, git pull pulls in changes --- bitbake core-image-core
Quoted 3 lines
The second "build", only the changed packages will be added to the image.
You you are using different build directories, then it doesn't re-use anything. The purpose of this is incremental software development within a single build directory.
You are still missing my point. The patch is broken if you change INC_RPM_IMAGE_GEN
Re: [PATCH 2/2] rootfs_rpm.bbclass: don't remove rpmlib when INC_RPM_IMAGE_GEN=1
Quoted 30 lines · Koen Kooi
Op 9 mei 2012, om 17:15 heeft Mark Hatle het volgende geschreven:
Quoted 27 lines · Koen Kooi
Quoted 7 lines
Op 9 mei 2012, om 16:52 heeft Mark Hatle het volgende geschreven:
Quoted 4 lines · Koen Kooi
Quoted 3 lines
Op 9 mei 2012, om 10:20 heeft Robert Yang het volgende geschreven:
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
Don't remove rpmlib when INC_RPM_IMAGE_GEN=1, otherwise the
list_installed_packageswould get nothing in the second build.What happens when I do a build, enable
INC_RPM_IMAGE_GENand do another build?The intention is that only the packages that have been changed/upgraded will be installed. Instead of the whole image being generated from scratch each time.
That's what
INC_RPM_IMAGE_GENdoes, yes. What I'm asking is how this patch addresses the sequence I outlined above.Explain what you mean by "build".
Using the following:
. ./oe-init-build-env build-test
<setup local.conf w/ rpm packaging andINC_RPM_IMAGE_GEN> bitbake core-image-core
--- time passes, git pull pulls in changes --- bitbake core-image-core
Quoted 5 lines
Quoted 3 lines
The second "build", only the changed packages will be added to the image.
You you are using different build directories, then it doesn't re-use anything. The purpose of this is incremental software development within a single build directory.
You are still missing my point. The patch is broken if you change
INC_RPM_IMAGE_GEN
I am absolutely missing your point.
I don't see what is wrong in the following, but I haven't tried it:
meta/classes/rootfs_rpm.bbclass EOF · line 161
Quoted 7 lines
meta/classes/rootfs_rpm.bbclass +5
-2- rm -rf ${IMAGE_ROOTFS}${rpmlibdir} - rm -rf ${IMAGE_ROOTFS}${opkglibdir} + # Don't remove the rpmlib when INC_RPM_IMAGE_GEN=1 + if [ "${INC_RPM_IMAGE_GEN}" != "1" ]; then + rm -rf ${IMAGE_ROOTFS}${rpmlibdir} + rm -rf ${IMAGE_ROOTFS}${opkglibdir} + fi
If the INC_RPM_IMAGE_GEN is not set to '1'. (Specifically '1' as opposed to just checking if it's set.) then it'll remove the existing image packaging database.
--Mark
Quoted 4 lines
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Re: [PATCH 2/2] rootfs_rpm.bbclass: don't remove rpmlib when INC_RPM_IMAGE_GEN=1
In reply to Koen KooiQuoted 30 lines · Koen Kooi
Op 9 mei 2012, om 17:15 heeft Mark Hatle het volgende geschreven:
Quoted 27 lines · Koen Kooi
Quoted 7 lines
Op 9 mei 2012, om 16:52 heeft Mark Hatle het volgende geschreven:
Quoted 4 lines · Koen Kooi
Quoted 3 lines
Op 9 mei 2012, om 10:20 heeft Robert Yang het volgende geschreven:
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
Don't remove rpmlib when INC_RPM_IMAGE_GEN=1, otherwise the
list_installed_packageswould get nothing in the second build.What happens when I do a build, enable
INC_RPM_IMAGE_GENand do another build?The intention is that only the packages that have been changed/upgraded will be installed. Instead of the whole image being generated from scratch each time.
That's what
INC_RPM_IMAGE_GENdoes, yes. What I'm asking is how this patch addresses the sequence I outlined above.Explain what you mean by "build".
Using the following:
. ./oe-init-build-env build-test
<setup local.conf w/ rpm packaging andINC_RPM_IMAGE_GEN> bitbake core-image-core
--- time passes, git pull pulls in changes --- bitbake core-image-core
Quoted 5 lines
Quoted 3 lines
The second "build", only the changed packages will be added to the image.
You you are using different build directories, then it doesn't re-use anything. The purpose of this is incremental software development within a single build directory.
You are still missing my point. The patch is broken if you change
INC_RPM_IMAGE_GEN
Only the core-image-minimal has this problem, other images won't remove the
database, the core-image-minimal removes the database to save disk space (
only saves quite a little), and I don't know whether there is a better solution
for the core-image-minimal, maybe move the database to ${T} rather than remove
it, and move it back to the rootfs directory at the early stage of do_rootfs, but I don't know whether this is worth or not since this is just used for saving the time at the development stage, as it had told:
# Incremental rpm image generation, the rootfs would be totally removed
# and re-created in the second generation by default, but with # INC_RPM_IMAGE_GEN = "1", the rpm based rootfs would be kept, and will # do update(remove/add some pkgs) on it. NOTE: This is not suggested # when you want to create a productive rootfs
// Robert
Quoted 5 lines
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Re: [PATCH 2/2] rootfs_rpm.bbclass: don't remove rpmlib when INC_RPM_IMAGE_GEN=1
Quoted 47 lines · Robert Yang
Quoted 37 lines · Koen Kooi
Op 9 mei 2012, om 17:15 heeft Mark Hatle het volgende geschreven:
Quoted 34 lines · Koen Kooi
Quoted 13 lines
Op 9 mei 2012, om 16:52 heeft Mark Hatle het volgende geschreven:
Quoted 10 lines · Koen Kooi
Quoted 8 lines
Op 9 mei 2012, om 10:20 heeft Robert Yang het volgende geschreven:
Quoted 2 lines
Don't remove rpmlib when INC_RPM_IMAGE_GEN=1, otherwise the
list_installed_packageswould get nothing in the second build.What happens when I do a build, enable
INC_RPM_IMAGE_GENand do another build?Quoted 6 lines
Quoted 3 lines
The intention is that only the packages that have been changed/upgraded will be installed. Instead of the whole image being generated from scratch each time.
That's what
INC_RPM_IMAGE_GENdoes, yes. What I'm asking is how this patch addresses the sequence I outlined above.Explain what you mean by "build".
Using the following:
. ./oe-init-build-env build-test
<setup local.conf w/ rpm packaging andINC_RPM_IMAGE_GEN> bitbake core-image-core
--- time passes, git pull pulls in changes --- bitbake core-image-coreThe second "build", only the changed packages will be added to the image.
Quoted 6 lines
Quoted 3 lines
You you are using different build directories, then it doesn't re-use anything. The purpose of this is incremental software development within a single build directory.
You are still missing my point. The patch is broken if you change
INC_RPM_IMAGE_GEN
Quoted 11 lines
Only the core-image-minimal has this problem, other images won't remove the database, the core-image-minimal removes the database to save disk space ( only saves quite a little), and I don't know whether there is a better solution
for the core-image-minimal, maybe move the database to ${T} rather than remove
it, and move it back to the rootfs directory at the early stage ofdo_rootfs, but I don't know whether this is worth or not since this is just used for
saving the time at the development stage, as it had told:
Robert,
The move might be the best solution since it will preserve it and make the behavior more predictable, ie if one starts with INC_RPM_IMAGE_GEN disabled and then enables it the package database will be available already. It would also preserve the space in the core-image-minimal by not keeping those database files in the image.
Please respin this patch.
Thanks
Sau!
Quoted 20 lines
# Incremental rpm image generation, the rootfs would be totally removed
# and re-created in the second generation by default, but with # INC_RPM_IMAGE_GEN = "1", the rpm based rootfs would be kept, and will # do update(remove/add some pkgs) on it. NOTE: This is not suggested # when you want to create a productive rootfs// Robert
Quoted 5 lines
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Re: [PATCH 2/2] rootfs_rpm.bbclass: don't remove rpmlib when INC_RPM_IMAGE_GEN=1
Quoted 59 lines · Saul Wold
Quoted 47 lines · Robert Yang
Quoted 37 lines · Koen Kooi
Op 9 mei 2012, om 17:15 heeft Mark Hatle het volgende geschreven:
Quoted 34 lines · Koen Kooi
Quoted 13 lines
Op 9 mei 2012, om 16:52 heeft Mark Hatle het volgende geschreven:
Quoted 10 lines · Koen Kooi
Quoted 8 lines
Op 9 mei 2012, om 10:20 heeft Robert Yang het volgende geschreven:
Quoted 2 lines
Don't remove rpmlib when INC_RPM_IMAGE_GEN=1, otherwise the
list_installed_packageswould get nothing in the second build.What happens when I do a build, enable
INC_RPM_IMAGE_GENand do another build?Quoted 6 lines
Quoted 3 lines
The intention is that only the packages that have been changed/upgraded will be installed. Instead of the whole image being generated from scratch each time.
That's what
INC_RPM_IMAGE_GENdoes, yes. What I'm asking is how this patch addresses the sequence I outlined above.Explain what you mean by "build".
Using the following:
. ./oe-init-build-env build-test
<setup local.conf w/ rpm packaging andINC_RPM_IMAGE_GEN> bitbake core-image-core
--- time passes, git pull pulls in changes --- bitbake core-image-coreThe second "build", only the changed packages will be added to the image.
Quoted 6 lines
Quoted 3 lines
You you are using different build directories, then it doesn't re-use anything. The purpose of this is incremental software development within a single build directory.
You are still missing my point. The patch is broken if you change
INC_RPM_IMAGE_GENQuoted 11 lines
Only the core-image-minimal has this problem, other images won't remove the database, the core-image-minimal removes the database to save disk space ( only saves quite a little), and I don't know whether there is a better solution
for the core-image-minimal, maybe move the database to ${T} rather than remove
it, and move it back to the rootfs directory at the early stage ofdo_rootfs, but I don't know whether this is worth or not since this is just used for
saving the time at the development stage, as it had told:
Quoted 8 lines
Robert,
The move might be the best solution since it will preserve it and make the behavior more predictable, ie if one starts with
INC_RPM_IMAGE_GENdisabled and then enables it the package database will be available already. It would also preserve the space in the core-image-minimal by not keeping those database files in the image.
Thanks Saul, I will send a V2 for it.
// Robert
Quoted 28 lines
Please respin this patch.
Thanks
Sau!Quoted 20 lines
# Incremental rpm image generation, the rootfs would be totally removed
# and re-created in the second generation by default, but with # INC_RPM_IMAGE_GEN = "1", the rpm based rootfs would be kept, and will # do update(remove/add some pkgs) on it. NOTE: This is not suggested # when you want to create a productive rootfs// Robert
Quoted 5 lines
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Re: [PATCH 2/2] rootfs_rpm.bbclass: don't remove rpmlib when INC_RPM_IMAGE_GEN=1
In reply to Mark HatleQuoted 6 lines · Mark Hatle
Quoted 5 lines
Quoted 5 lines · Koen Kooi
Quoted 3 lines
What happens when I do a build, enable
INC_RPM_IMAGE_GENand do another build?
Quoted 7 lines
Using the following:
. ./oe-init-build-env build-test
<setup local.conf w/ rpm packaging andINC_RPM_IMAGE_GEN> bitbake core-image-core
--- time passes, git pull pulls in changes --- bitbake core-image-core
That's not the same sequence. You were supposed to make first build
with INC_RPM_IMAGE_GEN unset, and set it for second build only.
- ML
Re: [PATCH 2/2] rootfs_rpm.bbclass: don't remove rpmlib when INC_RPM_IMAGE_GEN=1
Quoted 9 lines · Marko Lindqvist
Quoted 8 lines · Mark Hatle
Quoted 5 lines
Quoted 5 lines · Koen Kooi
Quoted 3 lines
What happens when I do a build, enable
INC_RPM_IMAGE_GENand do another build?Using the following:
Quoted 8 lines
Quoted 5 lines
. ./oe-init-build-env build-test
<setup local.conf w/ rpm packaging andINC_RPM_IMAGE_GEN> bitbake core-image-core
--- time passes, git pull pulls in changes --- bitbake core-image-coreThat's not the same sequence. You were supposed to make first build
withINC_RPM_IMAGE_GENunset, and set it for second build only.
If the value changes from one build to the next the behavior is undefined. It will probably work, but it's not guaranteed. (I'd need to look at the full implementation to verify that...)
--Mark
Quoted 7 lines
- ML
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core