package_manager.py/ootfs.py: support ipk incremental image generation
cover package_manager.py/ootfs.py: support ipk incremental image generation
Test Cases
---------------------------------------------
Case 1: Rebuild in place
1. Edit local.conf, enable multilib and ipk incremental image generation
...
46 MACHINE ?= "qemux86-64"
254 require conf/multilib.conf
255 MULTILIBS = "multilib:lib32"
256 DEFAULTTUNE_virtclass-multilib-lib32 = "x86"
257
258 IMAGE_INSTALL_append = " lib32-gzip"
260 IMAGE_FEATURES_append = " package-management ssh-server-dropbear " 262
263 INC_IPK_IMAGE_GEN = "1"
...
bb core-image-minimal
2. Edit local.conf, add some spaces in IMAGE_INSTALL_append vim local.conf
...
258 IMAGE_INSTALL_append = " lib32-gzip "
...
3. bb core-image-minimal, the image should not be rebuilt
vim log.do_rootfs:
...
16 NOTE: Package packagegroup-core-ssh-dropbear (1.0-r1) installed in root is up to date. 17 Package opkg-collateral (1.0-r2) installed in root is up to date.
18 Package run-postinsts (1.0-r9) installed in root is up to date.
19 Package packagegroup-core-boot (1.0-r11) installed in root is up to date. 20 Package opkg (1:0.2.0-r0) installed in root is up to date.
21 Package poky-feed-config-opkg (1.0-r2) installed in root is up to date.
...
53 NOTE: Package lib32-gzip (1.6-r7) installed in root is up to date.
...
vim installed_pkgs.txt
...
lib32-gzip x86
...
4. runqemu qemux86-64, ssh could work and lib32-gzip is installed On host:
runqemu qemux86-64
ssh [email protected] and run 'opkg-cl list-installed'
...
lib32-gzip - 1.6-r7
...
---------------------------------------------
Case 2: Decremental Rebuild
1. Edit local.conf, enable multilib and ipk incremental image generation
...
46 MACHINE ?= "qemux86-64"
254 require conf/multilib.conf
255 MULTILIBS = "multilib:lib32"
256 DEFAULTTUNE_virtclass-multilib-lib32 = "x86"
257
258 IMAGE_INSTALL_append = " lib32-gzip openssl"
260 IMAGE_FEATURES_append = " package-management ssh-server-dropbear " 262
263 INC_IPK_IMAGE_GEN = "1"
...
bb core-image-minimal
2. Edit local.conf, remove 'lib32-gzip openssl' vim local.conf
...
258 # IMAGE_INSTALL_append = " lib32-gzip openssl"
...
3. bb core-image-minimal, lib32-gzip and openssl should be removed. vim log.do_rootfs:
...
20 NOTE: Removing package lib32-eglibc from root...
21 Removing package lib32-gzip from root...
22 Removing package lib32-update-alternatives-opkg from root... 23 Removing package libcrypto1.0.0 from root...
24 Removing package libssl1.0.0 from root...
25 Removing package openssl from root...
26 Removing package openssl-conf from root...
...
31 Package opkg-collateral (1.0-r2) installed in root is up to date. 32 Package run-postinsts (1.0-r9) installed in root is up to date. 33 Package packagegroup-core-boot (1.0-r11) installed in root is up to date. 34 Package opkg (1:0.2.0-r0) installed in root is up to date.
35 Package poky-feed-config-opkg (1.0-r2) installed in root is up to date.
...
---------------------------------------------
Case 3: Incremental Rebuild
1. Edit local.conf, enable multilib and ipk incremental image generation
...
46 MACHINE ?= "qemux86-64"
254 require conf/multilib.conf
255 MULTILIBS = "multilib:lib32"
256 DEFAULTTUNE_virtclass-multilib-lib32 = "x86"
257
258 # IMAGE_INSTALL_append = " lib32-gzip openssl"
260 IMAGE_FEATURES_append = " package-management ssh-server-dropbear " 262
263 INC_IPK_IMAGE_GEN = "1"
...
bb core-image-minimal
2. Edit local.conf, append 'lib32-gzip openssl' to IMAGE_INSTALL vim local.conf
...
258 IMAGE_INSTALL_append = " lib32-gzip openssl"
...
3. bb core-image-minimal, lib32-gzip and openssl should be added.
vim log.do_rootfs:
...
20 NOTE: Package packagegroup-core-ssh-dropbear (1.0-r1) installed in root is up to date. 21 Package opkg-collateral (1.0-r2) installed in root is up to date.
22 Package run-postinsts (1.0-r9) installed in root is up to date.
23 Installing openssl (1.0.1e-r15.0) to root...
...
31 Package packagegroup-core-boot (1.0-r11) installed in root is up to date. 32 Package opkg (1:0.2.0-r0) installed in root is up to date.
33 Package poky-feed-config-opkg (1.0-r2) installed in root is up to date.
...
54 NOTE: Installing the following packages: lib32-gzip
...
---------------------------------------------
Case 4: Rebuild in place and Upgrade package
1. Edit local.conf, enable ipk incremental image generation
...
258 IMAGE_INSTALL_append = " gzip"
263 INC_IPK_IMAGE_GEN = "1"
...
bb core-image-minimal
2. Edit local.conf, upgrade gzip's PR
...
258 PR_pn-gzip = "r1"
...
3. bb core-image-minimal, gzip should be upgraded to r1
vim log.do_rootfs:
...
21 NOTE: Package packagegroup-core-ssh-dropbear (1.0-r1) installed in root is up to date. 22 Package opkg-collateral (1.0-r2) installed in root is up to date.
23 Package run-postinsts (1.0-r9) installed in root is up to date.
24 Package packagegroup-core-boot (1.0-r11) installed in root is up to date. 25 Package opkg (1:0.2.0-r0) installed in root is up to date.
26 Upgrading gzip on root from 1.6-r0 to 1.6-r1...
...
---------------------------------------------
Case 5: Test BAD_RECOMMENDATIONS
1. Edit local.conf, enable ipk incremental image generation
...
258 IMAGE_INSTALL_append = " util-linux"
263 INC_IPK_IMAGE_GEN = "1"
...
bb core-image-minimal
The util-linux-fdisk is in util-linux's RRECOMMENDS vim installed_pkgs.txt
...
41 util-linux-fdisk core2-64
...
2. Edit local.conf, add some spaces to IMAGE_INSTALL and add util-linux-fdisk to core-image-minimal's BAD_RECOMMENDATIONS
...
260 IMAGE_INSTALL_append = " util-linux "
265 BAD_RECOMMENDATIONS_pn-core-image-minimal += "util-linux-fdisk"
...
3. bb core-image-minimal, util-linux-fdisk should not be installed vim log.do_rootfs:
...
34 NOTE: Requested ignored recommendation util-linux-fdisk is not a package
...
119 util-linux: ignoring recommendation for util-linux-fdisk at user request 120 util-linux: unsatisfied recommendation for util-linux-fdisk
...
The util-linux-fdisk doesn't exist in installed_pkgs.txt any more.
---------------------------------------------
Case 5: Test NO_RECOMMENDATIONS
1. Edit local.conf, enable ipk incremental image generation
...
258 IMAGE_INSTALL_append = " util-linux"
263 INC_IPK_IMAGE_GEN = "1"
...
bb core-image-minimal
The util-linux-fdisk util-linux-cfdisk util-linux-sfdisk util-linux-mount util-linux-readprofile util-linux-mkfs are in util-linux's RRECOMMENDS
vim installed_pkgs.txt
...
4 util-linux-mkfs core2-64
18 util-linux-sfdisk core2-64
20 util-linux-mount core2-64
38 util-linux-cfdisk core2-64
41 util-linux-fdisk core2-64
46 util-linux-readprofile core2-64
...
2. Edit local.conf, add some spaces to IMAGE_INSTALL and assign core-image-minimal's NO_RECOMMENDATIONS with 1
...
260 IMAGE_INSTALL_append = " util-linux "
267 NO_RECOMMENDATIONS_pn-core-image-minimal = "1"
...
3. bb core-image-minimal, The six packages should not be installed vim log.do_rootfs:
...
34 NOTE: Requested ignored recommendation util-linux-fdisk is not a package
...
113 util-linux: ignoring recommendation for util-linux-sfdisk at user request 114 util-linux: unsatisfied recommendation for util-linux-sfdisk
115 util-linux: ignoring recommendation for util-linux-cfdisk at user request 116 util-linux: unsatisfied recommendation for util-linux-cfdisk
117 util-linux: ignoring recommendation for util-linux-mount at user request 118 util-linux: unsatisfied recommendation for util-linux-mount
119 util-linux: ignoring recommendation for util-linux-fdisk at user request 120 util-linux: unsatisfied recommendation for util-linux-fdisk
121 util-linux: ignoring recommendation for util-linux-mkfs at user request 122 util-linux: unsatisfied recommendation for util-linux-mkfs
123 util-linux: ignoring recommendation for util-linux-readprofile at user request 124 util-linux: unsatisfied recommendation for util-linux-readprofile
...
They don't exist in installed_pkgs.txt any more.
//Hongxu
The following changes since commit ac499f0b9f8a421a4e63d2de2a0b89f20c43d81c:
dev-manual: Fixed code block example for ARCHIVER. (2014-02-17 15:39:46 +0000)
are available in the git repository at:
git://git.pokylinux.org/poky-contrib hongxu/ipk-inc-gen
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=hongxu/ipk-inc-gen
Hongxu Jia (5):
package_manager.py: support ipk incremental image generation package_manager.py: tweak handle_bad_recommendations for ipk
incremental image generation
rootfs.py: support ipk incremental image generation rootfs.py: fix BAD_RECOMMENDATIONS for ipk incremental image
generation
rootfs.py: tweak _multilib_sanity_test for ipk incremental image
generation
meta/lib/oe/package_manager.py | 124 +++++++++++++++++++++++++++++++++++--- meta/lib/oe/rootfs.py | 134 ++++++++++++++++++++++++++++++++++++++--- 2 files changed, 242 insertions(+), 16 deletions(-)
Signature · 1 line
1.8.1.2
PATCH 1/5 package_manager.py: support ipk incremental image generation
While incremental image generation enabled, 'load_old_install_solution' is used to determine what we've got in the previous (existed) image and 'dump_install_solution' to determine what we need to install in the current image.
The 'backup_packaging_data' is used to back up the current opkg database.
The 'recovery_packaging_data' is used to recover the opkg database which backed up by the previous image creation.
Tweak 'remove' function in OpkgPM class, which the options for remove with dependencies was incorrect.
[YOCTO #1894]
--- meta/lib/oe/package_manager.py | 106 +++++++++++++++++++++++++++++++++++++++-- 1 file changed, 102 insertions(+), 4 deletions(-)
meta/lib/oe/package_manager.py +102 -4
@@ -934,12 +934,13 @@ class RpmPM(PackageManager): class OpkgPM(PackageManager): - def __init__(self, d, target_rootfs, config_file, archs): + def __init__(self, d, target_rootfs, config_file, archs, task_name='target'): super(OpkgPM, self).__init__(d) self.target_rootfs = target_rootfs self.config_file = config_file self.pkg_archs = archs + self.task_name = task_name self.deploy_dir = self.d.getVar("DEPLOY_DIR_IPK", True) self.deploy_lock_file = os.path.join(self.deploy_dir, "deploy.lock")
@@ -956,6 +957,13 @@ class OpkgPM(PackageManager): bb.utils.mkdirhier(self.opkg_dir) + self.solution_manifest = self.d.expand('${T}/saved/%s_solution' % + self.task_name) + self.saved_opkg_dir = self.d.expand('${T}/saved/%s' % self.task_name) + if not os.path.exists(self.d.expand('${T}/saved')): + bb.utils.mkdirhier(self.d.expand('${T}/saved')) + + if (self.d.getVar('BUILD_IMAGES_FROM_FEEDS', True) or "") != "1": self._create_config() else:
@@ -1075,7 +1083,9 @@ class OpkgPM(PackageManager): try: bb.note("Installing the following packages: %s" % ' '.join(pkgs)) - subprocess.check_output(cmd.split()) + bb.note(cmd) + output = subprocess.check_output(cmd.split()) + bb.note(output) except subprocess.CalledProcessError as e: (bb.fatal, bb.note)[attempt_only]("Unable to install packages. " "Command '%s' returned %d:\n%s" %
@@ -1083,14 +1093,16 @@ class OpkgPM(PackageManager): def remove(self, pkgs, with_dependencies=True): if with_dependencies: - cmd = "%s %s remove %s" % \ + cmd = "%s %s --force-depends --force-remove --force-removal-of-dependent-packages remove %s" % \ (self.opkg_cmd, self.opkg_args, ' '.join(pkgs)) else: cmd = "%s %s --force-depends remove %s" % \ (self.opkg_cmd, self.opkg_args, ' '.join(pkgs)) try: - subprocess.check_output(cmd.split()) + bb.note(cmd) + output = subprocess.check_output(cmd.split()) + bb.note(output) except subprocess.CalledProcessError as e: bb.fatal("Unable to remove packages. Command '%s' " "returned %d:\n%s" % (e.cmd, e.returncode, e.output))
@@ -1175,6 +1187,92 @@ class OpkgPM(PackageManager): else: status.write(line + "\n") + ''' + If incremental install, we need to determine what we've got, + what we need to add, and what to remove... + The dump_install_solution will dump and save the new install + solution. + ''' + def dump_install_solution(self, pkgs): + bb.note('creating new install solution for incremental install') + if len(pkgs) == 0: + return + + install_pkgs = list() + + # Create an temp dir as opkg root for simulating the installation + temp_rootfs = self.d.expand('${T}/opkg') + temp_opkg_dir = os.path.join(temp_rootfs, 'var/lib/opkg') + bb.utils.mkdirhier(temp_opkg_dir) + + opkg_args = "-f %s -o %s " % (self.config_file, temp_rootfs) + opkg_args += self.d.getVar("OPKG_ARGS", True) + + cmd = "%s %s update" % (self.opkg_cmd, + opkg_args) + try: + subprocess.check_output(cmd, shell=True) + except subprocess.CalledProcessError as e: + bb.note("Unable to dump install packages. Command '%s' " + "returned %d:\n%s" % (cmd, e.returncode, e.output)) + + # Simulate installation from zero + cmd = "%s %s --noaction install %s " % (self.opkg_cmd, + opkg_args, + ' '.join(pkgs)) + cmd += "| awk '/^Installing/{print $2}' " + cmd += "| sort -u -o %s" % self.solution_manifest + try: + subprocess.check_output(cmd, shell=True) + with open(self.solution_manifest, 'r') as manifest: + for pkg in manifest.read().split('\n'): + install_pkgs.append(pkg) + except subprocess.CalledProcessError as e: + bb.note("Unable to dump install packages. Command '%s' " + "returned %d:\n%s" % (cmd, e.returncode, e.output)) + + bb.utils.remove(temp_rootfs, True) + + return install_pkgs + + ''' + If incremental install, we need to determine what we've got, + what we need to add, and what to remove... + The load_old_install_solution will load the previous install + solution + ''' + def load_old_install_solution(self): + bb.note('load old install solution for incremental install') + installed_pkgs = list() + if not os.path.exists(self.solution_manifest): + bb.note('old install solution not exist') + return installed_pkgs + + with open(self.solution_manifest, 'r') as manifest: + for pkg in manifest.read().split('\n'): + installed_pkgs.append(pkg.strip()) + + return installed_pkgs + + def backup_packaging_data(self): + # Save the opkglib for increment rpm image generation + if os.path.exists(self.saved_opkg_dir): + bb.utils.remove(self.saved_opkg_dir, True) + shutil.copytree(self.opkg_dir, + self.saved_opkg_dir, + symlinks=True) + + def recovery_packaging_data(self): + # Move the opkglib back + if os.path.exists(self.saved_opkg_dir): + if os.path.exists(self.opkg_dir): + bb.utils.remove(self.opkg_dir, True) + + bb.note('Recovery packaging data') + shutil.copytree(self.saved_opkg_dir, + self.opkg_dir, + symlinks=True) + class DpkgPM(PackageManager): def __init__(self, d, target_rootfs, archs, base_archs, apt_conf_dir=None):
Signature · 1 line
1.8.1.2
Re: [PATCH 1/5] package_manager.py: support ipk incremental image generation
Quoted 84 lines · Hongxu Jia
While incremental image generation enabled, '
load_old_install_solution' is used to determine what we've got in the previous (existed) image and 'dump_install_solution' to determine what we need to install in the current image.The '
backup_packaging_data' is used to back up the current opkg database.The '
recovery_packaging_data' is used to recover the opkg database which backed up by the previous image creation.Tweak 'remove' function in OpkgPM class, which the options for remove with dependencies was incorrect.
[YOCTO #1894]
Signed-off-by Hongxu Jia <[email protected]>--- meta/lib/oe/package_manager.py | 106 +++++++++++++++++++++++++++++++++++++++-- 1 file changed, 102 insertions(+), 4 deletions(-)meta/lib/oe/package_manager.py +16
-4@@ -934,12 +934,13 @@ class RpmPM(PackageManager): class OpkgPM(PackageManager): - def __init__(self, d, target_rootfs, config_file, archs): + def __init__(self, d, target_rootfs, config_file, archs, task_name='target'): super(OpkgPM, self).__init__(d) self.target_rootfs = target_rootfs self.config_file = config_file self.pkg_archs = archs + self.task_name = task_name self.deploy_dir = self.d.getVar("DEPLOY_DIR_IPK", True) self.deploy_lock_file = os.path.join(self.deploy_dir, "deploy.lock")@@ -956,6 +957,13 @@ class OpkgPM(PackageManager): bb.utils.mkdirhier(self.opkg_dir) + self.solution_manifest = self.d.expand('${T}/saved/%s_solution' % + self.task_name) + self.saved_opkg_dir = self.d.expand('${T}/saved/%s' % self.task_name) + if not os.path.exists(self.d.expand('${T}/saved')): + bb.utils.mkdirhier(self.d.expand('${T}/saved')) + + if (self.d.getVar('BUILD_IMAGES_FROM_FEEDS', True) or "") != "1": self._create_config() else:@@ -1075,7 +1083,9 @@ class OpkgPM(PackageManager): try: bb.note("Installing the following packages: %s" % ' '.join(pkgs)) - subprocess.check_output(cmd.split()) + bb.note(cmd) + output = subprocess.check_output(cmd.split()) + bb.note(output) except subprocess.CalledProcessError as e: (bb.fatal, bb.note)[attempt_only]("Unable to install packages. " "Command '%s' returned %d:\n%s" %@@ -1083,14 +1093,16 @@ class OpkgPM(PackageManager): def remove(self, pkgs, with_dependencies=True): if with_dependencies: - cmd = "%s %s remove %s" % \ + cmd = "%s %s --force-depends --force-remove --force-removal-of-dependent-packages remove %s" % \ (self.opkg_cmd, self.opkg_args, ' '.join(pkgs)) else: cmd = "%s %s --force-depends remove %s" % \ (self.opkg_cmd, self.opkg_args, ' '.join(pkgs)) try: - subprocess.check_output(cmd.split()) + bb.note(cmd) + output = subprocess.check_output(cmd.split()) + bb.note(output) except subprocess.CalledProcessError as e: bb.fatal("Unable to remove packages. Command '%s' " "returned %d:\n%s" % (e.cmd, e.returncode, e.output))
meta/lib/oe/package_manager.py class OpkgPM(PackageManager): · line 1196
Quoted 10 lines
meta/lib/oe/package_manager.py +7
-0@@ -1175,6 +1187,92 @@ class OpkgPM(PackageManager): else: status.write(line + "\n") + ''' + If incremental install, we need to determine what we've got, + what we need to add, and what to remove... + The dump_install_solution will dump and save the new install + solution. + ''' + def dump_install_solution(self, pkgs):
Why not have this function in the Manifest class? We have an API in
place: Manifest.create_final() that should probably take care of anything related to manifest(s) creation after all packages are installed.
Quoted 18 lines
meta/lib/oe/package_manager.py +18
-0+ bb.note('creating new install solution for incremental install') + if len(pkgs) == 0: + return + + install_pkgs = list() + + # Create an temp dir as opkg root for simulating the installation + temp_rootfs = self.d.expand('${T}/opkg') + temp_opkg_dir = os.path.join(temp_rootfs, 'var/lib/opkg') + bb.utils.mkdirhier(temp_opkg_dir) + + opkg_args = "-f %s -o %s " % (self.config_file, temp_rootfs) + opkg_args += self.d.getVar("OPKG_ARGS", True) + + cmd = "%s %s update" % (self.opkg_cmd, + opkg_args) + try: + subprocess.check_output(cmd, shell=True)
meta/lib/oe/package_manager.py class OpkgPM(PackageManager): · line 1224
Quoted 10 lines
meta/lib/oe/package_manager.py +10
-0+ except subprocess.CalledProcessError as e: + bb.note("Unable to dump install packages. Command '%s' " + "returned %d:\n%s" % (cmd, e.returncode, e.output)) + + # Simulate installation from zero + cmd = "%s %s --noaction install %s " % (self.opkg_cmd, + opkg_args, + ' '.join(pkgs)) + cmd += "| awk '/^Installing/{print $2}' " + cmd += "| sort -u -o %s" % self.solution_manifest
Do we need to preserve this bash one-liner? Why not handle the command output using Python?
Quoted 10 lines
meta/lib/oe/package_manager.py +10
-0+ try: + subprocess.check_output(cmd, shell=True) + with open(self.solution_manifest, 'r') as manifest: + for pkg in manifest.read().split('\n'): + install_pkgs.append(pkg) + except subprocess.CalledProcessError as e: + bb.note("Unable to dump install packages. Command '%s' " + "returned %d:\n%s" % (cmd, e.returncode, e.output)) + + bb.utils.remove(temp_rootfs, True)
meta/lib/oe/package_manager.py class OpkgPM(PackageManager): · line 1244
Quoted 10 lines
meta/lib/oe/package_manager.py +10
-0+ + return install_pkgs + + ''' + If incremental install, we need to determine what we've got, + what we need to add, and what to remove... + The load_old_install_solution will load the previous install + solution + ''' + def load_old_install_solution(self):
Same here: why not put it in the Manifest class? As we already have the Manifest.parse_initial_manifest(), we can have something similar for the final_manifest (or install_solution, etc) parsing.
Quoted 4 lines
meta/lib/oe/package_manager.py +4
-0+ bb.note('load old install solution for incremental install') + installed_pkgs = list() + if not os.path.exists(self.solution_manifest): + bb.note('old install solution not exist')
meta/lib/oe/package_manager.py class OpkgPM(PackageManager): · line 1258
Quoted 10 lines
meta/lib/oe/package_manager.py +10
-0+ return installed_pkgs + + with open(self.solution_manifest, 'r') as manifest: + for pkg in manifest.read().split('\n'): + installed_pkgs.append(pkg.strip()) + + return installed_pkgs + + def backup_packaging_data(self): + # Save the opkglib for increment rpm image generation
s/rpm/opkg/
meta/lib/oe/package_manager.py class OpkgPM(PackageManager): · line 1265
Quoted 7 lines
meta/lib/oe/package_manager.py +7
-0+ if os.path.exists(self.saved_opkg_dir): + bb.utils.remove(self.saved_opkg_dir, True) + shutil.copytree(self.opkg_dir, + self.saved_opkg_dir, + symlinks=True) + + def recovery_packaging_data(self):
s/recovery/recover/ ?
Quoted 16 lines
meta/lib/oe/package_manager.py +10
-0+ # Move the opkglib back + if os.path.exists(self.saved_opkg_dir): + if os.path.exists(self.opkg_dir): + bb.utils.remove(self.opkg_dir, True) + + bb.note('Recovery packaging data') + shutil.copytree(self.saved_opkg_dir, + self.opkg_dir, + symlinks=True) + class DpkgPM(PackageManager): def __init__(self, d, target_rootfs, archs, base_archs, apt_conf_dir=None):Signature · 1 line
1.8.1.2
Re: [PATCH 1/5] package_manager.py: support ipk incremental image generation
Quoted 88 lines · Laurentiu Palcu
Quoted 87 lines · Hongxu Jia
While incremental image generation enabled, '
load_old_install_solution' is used to determine what we've got in the previous (existed) image and 'dump_install_solution' to determine what we need to install in the current image.The '
backup_packaging_data' is used to back up the current opkg database.The '
recovery_packaging_data' is used to recover the opkg database which backed up by the previous image creation.Tweak 'remove' function in OpkgPM class, which the options for remove with dependencies was incorrect.
[YOCTO #1894]
Signed-off-by Hongxu Jia <[email protected]>--- meta/lib/oe/package_manager.py | 106 +++++++++++++++++++++++++++++++++++++++-- 1 file changed, 102 insertions(+), 4 deletions(-)meta/lib/oe/package_manager.py +16
-4@@ -934,12 +934,13 @@ class RpmPM(PackageManager): class OpkgPM(PackageManager): - def __init__(self, d, target_rootfs, config_file, archs): + def __init__(self, d, target_rootfs, config_file, archs, task_name='target'): super(OpkgPM, self).__init__(d) self.target_rootfs = target_rootfs self.config_file = config_file self.pkg_archs = archs + self.task_name = task_name self.deploy_dir = self.d.getVar("DEPLOY_DIR_IPK", True) self.deploy_lock_file = os.path.join(self.deploy_dir, "deploy.lock")@@ -956,6 +957,13 @@ class OpkgPM(PackageManager): bb.utils.mkdirhier(self.opkg_dir) + self.solution_manifest = self.d.expand('${T}/saved/%s_solution' % + self.task_name) + self.saved_opkg_dir = self.d.expand('${T}/saved/%s' % self.task_name) + if not os.path.exists(self.d.expand('${T}/saved')): + bb.utils.mkdirhier(self.d.expand('${T}/saved')) + + if (self.d.getVar('BUILD_IMAGES_FROM_FEEDS', True) or "") != "1": self._create_config() else:@@ -1075,7 +1083,9 @@ class OpkgPM(PackageManager): try: bb.note("Installing the following packages: %s" % ' '.join(pkgs)) - subprocess.check_output(cmd.split()) + bb.note(cmd) + output = subprocess.check_output(cmd.split()) + bb.note(output) except subprocess.CalledProcessError as e: (bb.fatal, bb.note)[attempt_only]("Unable to install packages. " "Command '%s' returned %d:\n%s" %@@ -1083,14 +1093,16 @@ class OpkgPM(PackageManager): def remove(self, pkgs, with_dependencies=True): if with_dependencies: - cmd = "%s %s remove %s" % \ + cmd = "%s %s --force-depends --force-remove --force-removal-of-dependent-packages remove %s" % \ (self.opkg_cmd, self.opkg_args, ' '.join(pkgs)) else: cmd = "%s %s --force-depends remove %s" % \ (self.opkg_cmd, self.opkg_args, ' '.join(pkgs)) try: - subprocess.check_output(cmd.split()) + bb.note(cmd) + output = subprocess.check_output(cmd.split()) + bb.note(output) except subprocess.CalledProcessError as e: bb.fatal("Unable to remove packages. Command '%s' " "returned %d:\n%s" % (e.cmd, e.returncode, e.output))@@ -1175,6 +1187,92 @@ class OpkgPM(PackageManager): else: status.write(line + "\n")
Quoted 10 lines
meta/lib/oe/package_manager.py class OpkgPM(PackageManager): · line 1196
Quoted 7 lines
meta/lib/oe/package_manager.py +7
-0+ ''' + If incremental install, we need to determine what we've got, + what we need to add, and what to remove... + The dump_install_solution will dump and save the new install + solution. + ''' + def dump_install_solution(self, pkgs):Why not have this function in the Manifest class? We have an API in
place:Manifest.create_final()that should probably take care of anything related to manifest(s) creation after all packages are installed.
Got it,
The dump_install_solution creates a manifest with the packages that *will be* installed into the image, it is a dummy installation.
I will move 'dump_install_solution' to Manifest class as create_full.
Quoted 22 lines
Quoted 22 lines
meta/lib/oe/package_manager.py +22
-0+ bb.note('creating new install solution for incremental install') + if len(pkgs) == 0: + return + + install_pkgs = list() + + # Create an temp dir as opkg root for simulating the installation + temp_rootfs = self.d.expand('${T}/opkg') + temp_opkg_dir = os.path.join(temp_rootfs, 'var/lib/opkg') + bb.utils.mkdirhier(temp_opkg_dir) + + opkg_args = "-f %s -o %s " % (self.config_file, temp_rootfs) + opkg_args += self.d.getVar("OPKG_ARGS", True) + + cmd = "%s %s update" % (self.opkg_cmd, + opkg_args) + try: + subprocess.check_output(cmd, shell=True) + except subprocess.CalledProcessError as e: + bb.note("Unable to dump install packages. Command '%s' " + "returned %d:\n%s" % (cmd, e.returncode, e.output)) +
Quoted 8 lines
meta/lib/oe/package_manager.py class OpkgPM(PackageManager): · line 1224
Quoted 6 lines
meta/lib/oe/package_manager.py +6
-0+ # Simulate installation from zero + cmd = "%s %s --noaction install %s " % (self.opkg_cmd, + opkg_args, + ' '.join(pkgs)) + cmd += "| awk '/^Installing/{print $2}' " + cmd += "| sort -u -o %s" % self.solution_manifestDo we need to preserve this bash one-liner? Why not handle the command output using Python?
Agree, I will handle this in Python.
Quoted 15 lines
Quoted 15 lines
meta/lib/oe/package_manager.py +15
-0+ try: + subprocess.check_output(cmd, shell=True) + with open(self.solution_manifest, 'r') as manifest: + for pkg in manifest.read().split('\n'): + install_pkgs.append(pkg) + except subprocess.CalledProcessError as e: + bb.note("Unable to dump install packages. Command '%s' " + "returned %d:\n%s" % (cmd, e.returncode, e.output)) + + bb.utils.remove(temp_rootfs, True) + + return install_pkgs + + ''' + If incremental install, we need to determine what we've got,
Quoted 8 lines
meta/lib/oe/package_manager.py class OpkgPM(PackageManager): · line 1244
Quoted 5 lines
meta/lib/oe/package_manager.py +5
-0+ what we need to add, and what to remove... + The load_old_install_solution will load the previous install + solution + ''' + def load_old_install_solution(self):Same here: why not put it in the Manifest class? As we already have the
Manifest.parse_initial_manifest(), we can have something similar for thefinal_manifest(orinstall_solution, etc) parsing.
Agree, Manifest.parse_full_manifest() will be added
Quoted 7 lines
Quoted 7 lines
meta/lib/oe/package_manager.py +7
-0+ bb.note('load old install solution for incremental install') + installed_pkgs = list() + if not os.path.exists(self.solution_manifest): + bb.note('old install solution not exist') + return installed_pkgs + + with open(self.solution_manifest, 'r') as manifest:
Quoted 8 lines
meta/lib/oe/package_manager.py class OpkgPM(PackageManager): · line 1258
Quoted 7 lines
meta/lib/oe/package_manager.py +7
-0+ for pkg in manifest.read().split('\n'): + installed_pkgs.append(pkg.strip()) + + return installed_pkgs + + def backup_packaging_data(self): + # Save the opkglib for increment rpm image generations/rpm/opkg/
Sorry for the typo
Quoted 8 lines
meta/lib/oe/package_manager.py class OpkgPM(PackageManager): · line 1265
Quoted 7 lines
meta/lib/oe/package_manager.py +7
-0+ if os.path.exists(self.saved_opkg_dir): + bb.utils.remove(self.saved_opkg_dir, True) + shutil.copytree(self.opkg_dir, + self.saved_opkg_dir, + symlinks=True) + + def recovery_packaging_data(self):s/recovery/recover/ ?
Sorry for the typo
V2 incoming
//Hongxu
Quoted 16 lines
Quoted 16 lines
meta/lib/oe/package_manager.py +10
-0+ # Move the opkglib back + if os.path.exists(self.saved_opkg_dir): + if os.path.exists(self.opkg_dir): + bb.utils.remove(self.opkg_dir, True) + + bb.note('Recovery packaging data') + shutil.copytree(self.saved_opkg_dir, + self.opkg_dir, + symlinks=True) + class DpkgPM(PackageManager): def __init__(self, d, target_rootfs, archs, base_archs, apt_conf_dir=None):Signature · 1 line
1.8.1.2
PATCH 2/5 package_manager.py: tweak handle_bad_recommendations for ipk incremental image generation
In reply to Hongxu JiaLet opkg_dir be configurable in handle_bad_recommendations, so it could work on other root fs.
Invoke handle_bad_recommendations in dump_install_solution which simulates package install on other root fs.
--- meta/lib/oe/package_manager.py | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-)
meta/lib/oe/package_manager.py +13 -5
@@ -1157,12 +1157,19 @@ class OpkgPM(PackageManager): return output - def handle_bad_recommendations(self): - bad_recommendations = self.d.getVar("BAD_RECOMMENDATIONS", True) - if bad_recommendations is None: + def handle_bad_recommendations(self, opkg_dir=""): + bad_recommendations = self.d.getVar("BAD_RECOMMENDATIONS", True) or "" + if bad_recommendations.strip() == "": return - status_file = os.path.join(self.opkg_dir, "status") + if opkg_dir == "": + opkg_dir = self.opkg_dir + status_file = os.path.join(opkg_dir, "status") + + # If status file existed, it means the bad recommendations has already + # been handled + if os.path.exists(status_file): + return cmd = "%s %s info " % (self.opkg_cmd, self.opkg_args)
@@ -1177,7 +1184,7 @@ class OpkgPM(PackageManager): "returned %d:\n%s" % (pkg_info, e.returncode, e.output)) if output == "": - bb.note("Requested ignored recommendation $i is " + bb.note("Requested ignored recommendation %s is " "not a package" % pkg) continue
@@ -1204,6 +1211,7 @@ class OpkgPM(PackageManager): temp_rootfs = self.d.expand('${T}/opkg') temp_opkg_dir = os.path.join(temp_rootfs, 'var/lib/opkg') bb.utils.mkdirhier(temp_opkg_dir) + self.handle_bad_recommendations(temp_opkg_dir) opkg_args = "-f %s -o %s " % (self.config_file, temp_rootfs) opkg_args += self.d.getVar("OPKG_ARGS", True)
Signature · 1 line
1.8.1.2
PATCH 3/5 rootfs.py: support ipk incremental image generation
In reply to Hongxu JiaWhile incremental image generation enabled, if previous image existed, it restores the opkg database. Based on the previous image, it gets what need to remove for the current and remove them.
The newly added and upgraded packages will be done since opkg install invoked.
[YOCTO #1894]
--- meta/lib/oe/rootfs.py | 50 +++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 47 insertions(+), 3 deletions(-)
meta/lib/oe/rootfs.py +47 -3
@@ -438,13 +438,25 @@ class OpkgRootfs(Rootfs): def __init__(self, d, manifest_dir): super(OpkgRootfs, self).__init__(d) - bb.utils.remove(self.image_rootfs, True) - bb.utils.remove(self.d.getVar('MULTILIB_TEMP_ROOTFS', True), True) self.manifest = OpkgManifest(d, manifest_dir) self.opkg_conf = self.d.getVar("IPKGCONF_TARGET", True) self.pkg_archs = self.d.getVar("ALL_MULTILIB_PACKAGE_ARCHS", True) - self.pm = OpkgPM(d, self.image_rootfs, self.opkg_conf, self.pkg_archs) + self.inc_opkg_image_gen = self.d.getVar('INC_IPK_IMAGE_GEN', True) + if self.inc_opkg_image_gen != "1": + bb.utils.remove(self.image_rootfs, True) + self.pm = OpkgPM(d, + self.image_rootfs, + self.opkg_conf, + self.pkg_archs) + else: + self.pm = OpkgPM(d, + self.image_rootfs, + self.opkg_conf, + self.pkg_archs) + self.pm.recovery_packaging_data() + + bb.utils.remove(self.d.getVar('MULTILIB_TEMP_ROOTFS', True), True) """ This function was reused from the old implementation.
@@ -508,6 +520,32 @@ class OpkgRootfs(Rootfs): self._multilib_sanity_test(dirs) + ''' + While ipk incremental image generation is enabled, it will remove the + unneeded pkgs by comparing the new install solution manifest and the + old installed manifest. + ''' + def _create_incremental(self, pkgs_initial_install): + if self.inc_opkg_image_gen == "1": + + pkgs_to_install = list() + for pkg_type in pkgs_initial_install: + pkgs_to_install += pkgs_initial_install[pkg_type] + + installed_manifest = self.pm.load_old_install_solution() + solution_manifest = self.pm.dump_install_solution(pkgs_to_install) + + pkg_to_remove = list() + for pkg in installed_manifest: + if pkg not in solution_manifest: + pkg_to_remove.append(pkg) + + self.pm.update() + + if pkg_to_remove != []: + bb.note('incremental removed: %s' % ' '.join(pkg_to_remove)) + self.pm.remove(pkg_to_remove) + def _create(self): pkgs_to_install = self.manifest.parse_initial_manifest() opkg_pre_process_cmds = self.d.getVar('OPKG_PREPROCESS_COMMANDS', True)
@@ -518,6 +556,9 @@ class OpkgRootfs(Rootfs): if (self.d.getVar('BUILD_IMAGES_FROM_FEEDS', True) or "") != "1": self.pm.write_index() + if self.inc_opkg_image_gen == "1": + self._create_incremental(pkgs_to_install) + execute_pre_post_process(self.d, opkg_pre_process_cmds) self.pm.update()
@@ -540,6 +581,9 @@ class OpkgRootfs(Rootfs): execute_pre_post_process(self.d, opkg_post_process_cmds) execute_pre_post_process(self.d, rootfs_post_install_cmds) + if self.inc_opkg_image_gen == "1": + self.pm.backup_packaging_data() + def _get_delayed_postinsts(self): pkg_list = [] status_file = os.path.join(self.image_rootfs,
Signature · 1 line
1.8.1.2
PATCH 4/5 rootfs.py: fix BAD_RECOMMENDATIONS for ipk incremental image generation
In reply to Hongxu JiaWhile incremental image generation enabled and the previous image existed, if BAD_RECOMMENDATIONS is changed, the operation on the existed image is complicated, so remove the existed image in this situation.
The same with PACKAGE_EXCLUDE and NO_RECOMMENDATIONS.
--- meta/lib/oe/rootfs.py | 30 +++++++++++++++++++++++++++++- 1 file changed, 29 insertions(+), 1 deletion(-)
meta/lib/oe/rootfs.py +29 -1
@@ -443,7 +443,7 @@ class OpkgRootfs(Rootfs): self.pkg_archs = self.d.getVar("ALL_MULTILIB_PACKAGE_ARCHS", True) self.inc_opkg_image_gen = self.d.getVar('INC_IPK_IMAGE_GEN', True) - if self.inc_opkg_image_gen != "1": + if self._remove_existed_image(): bb.utils.remove(self.image_rootfs, True) self.pm = OpkgPM(d, self.image_rootfs,
@@ -546,6 +546,34 @@ class OpkgRootfs(Rootfs): bb.note('incremental removed: %s' % ' '.join(pkg_to_remove)) self.pm.remove(pkg_to_remove) + ''' + Compare with previous (existed) image creation, if some conditions + triggered, the previous (existed) image should be removed. + The conditions included any of 'PACKAGE_EXCLUDE, NO_RECOMMENDATIONS + and BAD_RECOMMENDATIONS' has been changed. + ''' + def _remove_existed_image(self): + # Incremental image creation is not enable + if self.inc_opkg_image_gen != "1": + return True + + vars_list_dir = self.d.expand('${T}/vars_list') + + old_vars_list = "" + if os.path.exists(vars_list_dir): + old_vars_list = open(vars_list_dir, 'r+').read() + + new_vars_list = '%s:%s:%s\n' % \ + ((self.d.getVar('BAD_RECOMMENDATIONS', True) or '').strip(), + (self.d.getVar('NO_RECOMMENDATIONS', True) or '').strip(), + (self.d.getVar('PACKAGE_EXCLUDE', True) or '').strip()) + open(vars_list_dir, 'w+').write(new_vars_list) + + if old_vars_list != new_vars_list: + return True + + return False + def _create(self): pkgs_to_install = self.manifest.parse_initial_manifest() opkg_pre_process_cmds = self.d.getVar('OPKG_PREPROCESS_COMMANDS', True)
Signature · 1 line
1.8.1.2
Re: [PATCH 4/5] rootfs.py: fix BAD_RECOMMENDATIONS for ipk incremental image generation
Quoted 26 lines · Hongxu Jia
While incremental image generation enabled and the previous image existed, if
BAD_RECOMMENDATIONSis changed, the operation on the existed image is complicated, so remove the existed image in this situation.The same with
PACKAGE_EXCLUDEandNO_RECOMMENDATIONS.[YOCTO #1894]Signed-off-by Hongxu Jia <[email protected]>--- meta/lib/oe/rootfs.py | 30 +++++++++++++++++++++++++++++- 1 file changed, 29 insertions(+), 1 deletion(-)meta/lib/oe/rootfs.py +1
-1@@ -443,7 +443,7 @@ class OpkgRootfs(Rootfs): self.pkg_archs = self.d.getVar("ALL_MULTILIB_PACKAGE_ARCHS", True) self.inc_opkg_image_gen = self.d.getVar('INC_IPK_IMAGE_GEN', True) - if self.inc_opkg_image_gen != "1": + if self._remove_existed_image(): bb.utils.remove(self.image_rootfs, True) self.pm = OpkgPM(d, self.image_rootfs,
meta/lib/oe/rootfs.py class OpkgRootfs(Rootfs): · line 555
Quoted 10 lines
meta/lib/oe/rootfs.py +7
-0@@ -546,6 +546,34 @@ class OpkgRootfs(Rootfs): bb.note('incremental removed: %s' % ' '.join(pkg_to_remove)) self.pm.remove(pkg_to_remove) + ''' + Compare with previous (existed) image creation, if some conditions + triggered, the previous (existed) image should be removed. + The conditions included any of 'PACKAGE_EXCLUDE, NO_RECOMMENDATIONS + and BAD_RECOMMENDATIONS' has been changed. + ''' + def _remove_existed_image(self):
s/existed/existing/
Also in the comment above.
meta/lib/oe/rootfs.py class OpkgRootfs(Rootfs): · line 560
Quoted 5 lines
meta/lib/oe/rootfs.py +5
-0+ # Incremental image creation is not enable + if self.inc_opkg_image_gen != "1": + return True + + vars_list_dir = self.d.expand('${T}/vars_list')
s/vars_list_dir/vars_list_file/ in the line above and the code below. Leaving this variable with a 'dir' suffix while we're opening a file for reading/writing, is a little misleading.
Quoted 22 lines
meta/lib/oe/rootfs.py +16
-0+ + old_vars_list = "" + if os.path.exists(vars_list_dir): + old_vars_list = open(vars_list_dir, 'r+').read() + + new_vars_list = '%s:%s:%s\n' % \ + ((self.d.getVar('BAD_RECOMMENDATIONS', True) or '').strip(), + (self.d.getVar('NO_RECOMMENDATIONS', True) or '').strip(), + (self.d.getVar('PACKAGE_EXCLUDE', True) or '').strip()) + open(vars_list_dir, 'w+').write(new_vars_list) + + if old_vars_list != new_vars_list: + return True + + return False + def _create(self): pkgs_to_install = self.manifest.parse_initial_manifest() opkg_pre_process_cmds = self.d.getVar('OPKG_PREPROCESS_COMMANDS', True)Signature · 1 line
1.8.1.2
Re: [PATCH 4/5] rootfs.py: fix BAD_RECOMMENDATIONS for ipk incremental image generation
Quoted 39 lines · Laurentiu Palcu
Quoted 26 lines · Hongxu Jia
While incremental image generation enabled and the previous image existed, if
BAD_RECOMMENDATIONSis changed, the operation on the existed image is complicated, so remove the existed image in this situation.The same with
PACKAGE_EXCLUDEandNO_RECOMMENDATIONS.[YOCTO #1894]Signed-off-by Hongxu Jia <[email protected]>--- meta/lib/oe/rootfs.py | 30 +++++++++++++++++++++++++++++- 1 file changed, 29 insertions(+), 1 deletion(-)meta/lib/oe/rootfs.py +1
-1@@ -443,7 +443,7 @@ class OpkgRootfs(Rootfs): self.pkg_archs = self.d.getVar("ALL_MULTILIB_PACKAGE_ARCHS", True) self.inc_opkg_image_gen = self.d.getVar('INC_IPK_IMAGE_GEN', True) - if self.inc_opkg_image_gen != "1": + if self._remove_existed_image(): bb.utils.remove(self.image_rootfs, True) self.pm = OpkgPM(d, self.image_rootfs,meta/lib/oe/rootfs.py class OpkgRootfs(Rootfs): · line 555
Quoted 10 lines
meta/lib/oe/rootfs.py +7
-0@@ -546,6 +546,34 @@ class OpkgRootfs(Rootfs): bb.note('incremental removed: %s' % ' '.join(pkg_to_remove)) self.pm.remove(pkg_to_remove) + ''' + Compare with previous (existed) image creation, if some conditions + triggered, the previous (existed) image should be removed. + The conditions included any of 'PACKAGE_EXCLUDE, NO_RECOMMENDATIONS + and BAD_RECOMMENDATIONS' has been changed. + ''' + def _remove_existed_image(self):s/existed/existing/
Quoted 1 line
Also in the comment above.
Sorry for the typo
Quoted 9 lines
meta/lib/oe/rootfs.py class OpkgRootfs(Rootfs): · line 560
Quoted 6 lines
meta/lib/oe/rootfs.py +5
-0+ # Incremental image creation is not enable + if self.inc_opkg_image_gen != "1": + return True + + vars_list_dir = self.d.expand('${T}/vars_list')s/vars_list_dir/vars_list_file/ in the line above and the code below. Leaving this variable with a 'dir' suffix while we're opening a file for reading/writing, is a little misleading.
Sorry for the misleading
V2 incoming
//Hongxu
Quoted 22 lines
Quoted 22 lines
meta/lib/oe/rootfs.py +16
-0+ + old_vars_list = "" + if os.path.exists(vars_list_dir): + old_vars_list = open(vars_list_dir, 'r+').read() + + new_vars_list = '%s:%s:%s\n' % \ + ((self.d.getVar('BAD_RECOMMENDATIONS', True) or '').strip(), + (self.d.getVar('NO_RECOMMENDATIONS', True) or '').strip(), + (self.d.getVar('PACKAGE_EXCLUDE', True) or '').strip()) + open(vars_list_dir, 'w+').write(new_vars_list) + + if old_vars_list != new_vars_list: + return True + + return False + def _create(self): pkgs_to_install = self.manifest.parse_initial_manifest() opkg_pre_process_cmds = self.d.getVar('OPKG_PREPROCESS_COMMANDS', True)Signature · 1 line
1.8.1.2
PATCH 5/5 rootfs.py: tweak _multilib_sanity_test for ipk incremental image generation
In reply to Hongxu JiaThe _multilib_sanity_test installs multilib packages in a temporary root fs, and compare with the current image to figure out duplicated file that comes from different packages.
While incremental image generation enabled and the previous image existed, there was a Multilib check error:
...
ERROR: Multilib check error: duplicate files tmp/work/qemux86_64-poky- linux/core-image-minimal/1.0-r0/multilib/lib32/lib/libc.so.6 tmp/work/ qemux86_64-poky-linux/core-image-minimal/1.0-r0/rootfs/lib/libc.so.6 is not the same
...
The reason is the file in current image has been prelinked in previous image generation and the file in a temporary root fs is not prelinked, even though the files come from the same package, the Multilib check considers they are different.
--- meta/lib/oe/rootfs.py | 56 +++++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 52 insertions(+), 4 deletions(-)
meta/lib/oe/rootfs.py +52 -4
@@ -3,6 +3,8 @@ from oe.utils import execute_pre_post_process from oe.utils import contains as base_contains from oe.package_manager import * from oe.manifest import * +import oe.path +import filecmp import shutil import os import subprocess
@@ -458,13 +460,61 @@ class OpkgRootfs(Rootfs): bb.utils.remove(self.d.getVar('MULTILIB_TEMP_ROOTFS', True), True) + def _prelink_file(self, root_dir, filename): + bb.note('prelink %s in %s' % (filename, root_dir)) + prelink_cfg = oe.path.join(root_dir, + self.d.expand('${sysconfdir}/prelink.conf')) + if not os.path.exists(prelink_cfg): + shutil.copy(self.d.expand('${STAGING_DIR_NATIVE}${sysconfdir_native}/prelink.conf'), + prelink_cfg) + + cmd_prelink = self.d.expand('${STAGING_DIR_NATIVE}${sbindir_native}/prelink') + self._exec_shell_cmd([cmd_prelink, + '--root', + root_dir, + '-amR', + '-N', + '-c', + self.d.expand('${sysconfdir}/prelink.conf')]) + + ''' + Compare two files with the same key twice to see if they came + from the same package. If they are not same, they are duplicated + and come from different packages. + 1st: Comapre them directly; + 2nd: While incremental image creation is enabled, one of the + files could be probaly prelinked in the previous image + creation and the file has been changed, so we need to + prelink the other one and compare them. + ''' + def _file_duplicate(self, key, f1, f2): + + if not os.path.exists(f1) or not os.path.exists(f2): + return False + + # f1 is the same with f2, both of them were not prelinked + if filecmp.cmp(f1, f2): + return False + + if self.image_rootfs not in f1: + self._prelink_file(f1.replace(key, ''), f1) + + if self.image_rootfs not in f2: + self._prelink_file(f2.replace(key, ''), f2) + + # f1 is the same with f2, both of them were prelinked + if filecmp.cmp(f1, f2): + return False + + # Duplicated + return True + """ This function was reused from the old implementation. See commit: "image.bbclass: Added variables for multilib support." by Lianhao Lu. """ def _multilib_sanity_test(self, dirs): - import filecmp allow_replace = self.d.getVar("MULTILIBRE_ALLOW_REP", True) if allow_replace is None:
@@ -486,9 +536,7 @@ class OpkgRootfs(Rootfs): if allow_rep.match(key): valid = True else: - if os.path.exists(files[key]) and \ - os.path.exists(item) and \ - not filecmp.cmp(files[key], item): + if self._file_duplicate(key, files[key], item): valid = False bb.fatal("%s duplicate files %s %s is not the same\n" % (error_prompt, item, files[key]))
Signature · 1 line
1.8.1.2
Re: [PATCH 5/5] rootfs.py: tweak _multilib_sanity_test for ipk incremental image generation
Quoted 52 lines · Hongxu Jia
The
_multilib_sanity_testinstalls multilib packages in a temporary root fs, and compare with the current image to figure out duplicated file that comes from different packages.While incremental image generation enabled and the previous image existed, there was a Multilib check error:
...
ERROR: Multilib check error: duplicate files tmp/work/qemux86_64-poky- linux/core-image-minimal/1.0-r0/multilib/lib32/lib/libc.so.6 tmp/work/ qemux86_64-poky-linux/core-image-minimal/1.0-r0/rootfs/lib/libc.so.6 is not the same
...The reason is the file in current image has been prelinked in previous image generation and the file in a temporary root fs is not prelinked, even though the files come from the same package, the Multilib check considers they are different.
[YOCTO #1894]Signed-off-by Hongxu Jia <[email protected]>--- meta/lib/oe/rootfs.py | 56 +++++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 52 insertions(+), 4 deletions(-)meta/lib/oe/rootfs.py +13
-0@@ -3,6 +3,8 @@ from oe.utils import execute_pre_post_process from oe.utils import contains as base_contains from oe.package_manager import * from oe.manifest import * +import oe.path +import filecmp import shutil import os import subprocess@@ -458,13 +460,61 @@ class OpkgRootfs(Rootfs): bb.utils.remove(self.d.getVar('MULTILIB_TEMP_ROOTFS', True), True) + def _prelink_file(self, root_dir, filename): + bb.note('prelink %s in %s' % (filename, root_dir)) + prelink_cfg = oe.path.join(root_dir, + self.d.expand('${sysconfdir}/prelink.conf')) + if not os.path.exists(prelink_cfg): + shutil.copy(self.d.expand('${STAGING_DIR_NATIVE}${sysconfdir_native}/prelink.conf'), + prelink_cfg) + + cmd_prelink = self.d.expand('${STAGING_DIR_NATIVE}${sbindir_native}/prelink') + self._exec_shell_cmd([cmd_prelink, + '--root',
meta/lib/oe/rootfs.py class OpkgRootfs(Rootfs): · line 483
Quoted 10 lines
meta/lib/oe/rootfs.py +10
-0+ root_dir, + '-amR', + '-N', + '-c', + self.d.expand('${sysconfdir}/prelink.conf')]) + + ''' + Compare two files with the same key twice to see if they came + from the same package. If they are not same, they are duplicated + and come from different packages.
I'm kind of confused by this comment. Doesn't same = duplicate? There might be a small confusion of terms here because the function's behavior is not as the name implies.
meta/lib/oe/rootfs.py class OpkgRootfs(Rootfs): · line 490
Quoted 7 lines
meta/lib/oe/rootfs.py +7
-0+ 1st: Comapre them directly; + 2nd: While incremental image creation is enabled, one of the + files could be probaly prelinked in the previous image + creation and the file has been changed, so we need to + prelink the other one and compare them. + ''' + def _file_duplicate(self, key, f1, f2):
Shouldn't be better to rename this function to something else in order to avoid confusion? Let's say: _files_are_equal() ?
meta/lib/oe/rootfs.py class OpkgRootfs(Rootfs): · line 497
Quoted 7 lines
meta/lib/oe/rootfs.py +7
-0+ + if not os.path.exists(f1) or not os.path.exists(f2): + return False + + # f1 is the same with f2, both of them were not prelinked + if filecmp.cmp(f1, f2): + return False
filecmp.cmp() returns True if files are equal. Hence the confusion:_file_duplicate() returns False here, if files are equal... I think the logic is a little bit the other way around! :)
meta/lib/oe/rootfs.py class OpkgRootfs(Rootfs): · line 510
Quoted 13 lines
meta/lib/oe/rootfs.py +13
-0+ + if self.image_rootfs not in f1: + self._prelink_file(f1.replace(key, ''), f1) + + if self.image_rootfs not in f2: + self._prelink_file(f2.replace(key, ''), f2) + + # f1 is the same with f2, both of them were prelinked + if filecmp.cmp(f1, f2): + return False + + # Duplicated + return True
here the return value matches the comment and the function name! :)
Quoted 25 lines
meta/lib/oe/rootfs.py +2
-4+ """ This function was reused from the old implementation. See commit: "image.bbclass: Added variables for multilib support." by Lianhao Lu. """ def _multilib_sanity_test(self, dirs): - import filecmp allow_replace = self.d.getVar("MULTILIBRE_ALLOW_REP", True) if allow_replace is None:@@ -486,9 +536,7 @@ class OpkgRootfs(Rootfs): if allow_rep.match(key): valid = True else: - if os.path.exists(files[key]) and \ - os.path.exists(item) and \ - not filecmp.cmp(files[key], item): + if self._file_duplicate(key, files[key], item): valid = False bb.fatal("%s duplicate files %s %s is not the same\n" % (error_prompt, item, files[key]))Signature · 1 line
1.8.1.2
Re: [PATCH 5/5] rootfs.py: tweak _multilib_sanity_test for ipk incremental image generation
I will rename _file_duplicate with _file_equal, and do the necessary ajustment to avoid confusions.
Thank you for pointing out
V2 incoming
//Hongxu
Quoted 127 lines · Laurentiu Palcu
Quoted 52 lines · Hongxu Jia
The
_multilib_sanity_testinstalls multilib packages in a temporary root fs, and compare with the current image to figure out duplicated file that comes from different packages.While incremental image generation enabled and the previous image existed, there was a Multilib check error:
...
ERROR: Multilib check error: duplicate files tmp/work/qemux86_64-poky- linux/core-image-minimal/1.0-r0/multilib/lib32/lib/libc.so.6 tmp/work/ qemux86_64-poky-linux/core-image-minimal/1.0-r0/rootfs/lib/libc.so.6 is not the same
...The reason is the file in current image has been prelinked in previous image generation and the file in a temporary root fs is not prelinked, even though the files come from the same package, the Multilib check considers they are different.
[YOCTO #1894]Signed-off-by Hongxu Jia <[email protected]>--- meta/lib/oe/rootfs.py | 56 +++++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 52 insertions(+), 4 deletions(-)meta/lib/oe/rootfs.py +13
-0@@ -3,6 +3,8 @@ from oe.utils import execute_pre_post_process from oe.utils import contains as base_contains from oe.package_manager import * from oe.manifest import * +import oe.path +import filecmp import shutil import os import subprocess@@ -458,13 +460,61 @@ class OpkgRootfs(Rootfs): bb.utils.remove(self.d.getVar('MULTILIB_TEMP_ROOTFS', True), True) + def _prelink_file(self, root_dir, filename): + bb.note('prelink %s in %s' % (filename, root_dir)) + prelink_cfg = oe.path.join(root_dir, + self.d.expand('${sysconfdir}/prelink.conf')) + if not os.path.exists(prelink_cfg): + shutil.copy(self.d.expand('${STAGING_DIR_NATIVE}${sysconfdir_native}/prelink.conf'), + prelink_cfg) + + cmd_prelink = self.d.expand('${STAGING_DIR_NATIVE}${sbindir_native}/prelink') + self._exec_shell_cmd([cmd_prelink, + '--root',meta/lib/oe/rootfs.py class OpkgRootfs(Rootfs): · line 483
Quoted 10 lines
meta/lib/oe/rootfs.py +10
-0+ root_dir, + '-amR', + '-N', + '-c', + self.d.expand('${sysconfdir}/prelink.conf')]) + + ''' + Compare two files with the same key twice to see if they came + from the same package. If they are not same, they are duplicated + and come from different packages.I'm kind of confused by this comment. Doesn't same = duplicate? There might be a small confusion of terms here because the function's behavior is not as the name implies.
meta/lib/oe/rootfs.py class OpkgRootfs(Rootfs): · line 490
Quoted 7 lines
meta/lib/oe/rootfs.py +7
-0+ 1st: Comapre them directly; + 2nd: While incremental image creation is enabled, one of the + files could be probaly prelinked in the previous image + creation and the file has been changed, so we need to + prelink the other one and compare them. + ''' + def _file_duplicate(self, key, f1, f2):Shouldn't be better to rename this function to something else in order to avoid confusion? Let's say:
_files_are_equal()?meta/lib/oe/rootfs.py class OpkgRootfs(Rootfs): · line 497
Quoted 7 lines
meta/lib/oe/rootfs.py +7
-0+ + if not os.path.exists(f1) or not os.path.exists(f2): + return False + + # f1 is the same with f2, both of them were not prelinked + if filecmp.cmp(f1, f2): + return False
filecmp.cmp()returns True if files are equal. Hence the confusion:_file_duplicate()returns False here, if files are equal... I think the logic is a little bit the other way around! :)meta/lib/oe/rootfs.py class OpkgRootfs(Rootfs): · line 510
Quoted 13 lines
meta/lib/oe/rootfs.py +13
-0+ + if self.image_rootfs not in f1: + self._prelink_file(f1.replace(key, ''), f1) + + if self.image_rootfs not in f2: + self._prelink_file(f2.replace(key, ''), f2) + + # f1 is the same with f2, both of them were prelinked + if filecmp.cmp(f1, f2): + return False + + # Duplicated + return Truehere the return value matches the comment and the function name! :)
Quoted 25 lines
meta/lib/oe/rootfs.py +2
-4+ """ This function was reused from the old implementation. See commit: "image.bbclass: Added variables for multilib support." by Lianhao Lu. """ def _multilib_sanity_test(self, dirs): - import filecmp allow_replace = self.d.getVar("MULTILIBRE_ALLOW_REP", True) if allow_replace is None:@@ -486,9 +536,7 @@ class OpkgRootfs(Rootfs): if allow_rep.match(key): valid = True else: - if os.path.exists(files[key]) and \ - os.path.exists(item) and \ - not filecmp.cmp(files[key], item): + if self._file_duplicate(key, files[key], item): valid = False bb.fatal("%s duplicate files %s %s is not the same\n" % (error_prompt, item, files[key]))Signature · 1 line
1.8.1.2