Skip to contentSkip to conversation

package_manager.py/ootfs.py: support ipk incremental image generation

PATCH Superseded Hongxu Jia 2 participants 12 messages Feb 18, 2014 – Feb 19, 2014

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Hongxu Jia

author
#0 lore raw

cover package_manager.py/ootfs.py: support ipk incremental image generation

Test Cases

---------------------------------------------
Case 1: Rebuild in place
1. Edit local.conf, enable multilib and ipk incremental image generation
...
46 MACHINE ?= "qemux86-64"
254 require conf/multilib.conf
255 MULTILIBS = "multilib:lib32"
256 DEFAULTTUNE_virtclass-multilib-lib32 = "x86"
257
258 IMAGE_INSTALL_append = " lib32-gzip"
260 IMAGE_FEATURES_append = " package-management ssh-server-dropbear " 262
263 INC_IPK_IMAGE_GEN = "1"
...
bb core-image-minimal

2. Edit local.conf, add some spaces in IMAGE_INSTALL_append vim local.conf
...
258 IMAGE_INSTALL_append = " lib32-gzip "
...

3. bb core-image-minimal, the image should not be rebuilt
vim log.do_rootfs:
...
16 NOTE: Package packagegroup-core-ssh-dropbear (1.0-r1) installed in root is up to date. 17 Package opkg-collateral (1.0-r2) installed in root is up to date.
18 Package run-postinsts (1.0-r9) installed in root is up to date.
19 Package packagegroup-core-boot (1.0-r11) installed in root is up to date. 20 Package opkg (1:0.2.0-r0) installed in root is up to date.
21 Package poky-feed-config-opkg (1.0-r2) installed in root is up to date.
...
53 NOTE: Package lib32-gzip (1.6-r7) installed in root is up to date.
...

vim installed_pkgs.txt
...
lib32-gzip x86
...

4. runqemu qemux86-64, ssh could work and lib32-gzip is installed On host:
runqemu qemux86-64
ssh [email protected] and run 'opkg-cl list-installed'
...
lib32-gzip - 1.6-r7
...

---------------------------------------------
Case 2: Decremental Rebuild
1. Edit local.conf, enable multilib and ipk incremental image generation
...
46 MACHINE ?= "qemux86-64"
254 require conf/multilib.conf
255 MULTILIBS = "multilib:lib32"
256 DEFAULTTUNE_virtclass-multilib-lib32 = "x86"
257
258 IMAGE_INSTALL_append = " lib32-gzip openssl"
260 IMAGE_FEATURES_append = " package-management ssh-server-dropbear " 262
263 INC_IPK_IMAGE_GEN = "1"
...
bb core-image-minimal

2. Edit local.conf, remove 'lib32-gzip openssl' vim local.conf
...
258 # IMAGE_INSTALL_append = " lib32-gzip openssl"
...

3. bb core-image-minimal, lib32-gzip and openssl should be removed. vim log.do_rootfs:
...
20 NOTE: Removing package lib32-eglibc from root...
21 Removing package lib32-gzip from root...
22 Removing package lib32-update-alternatives-opkg from root... 23 Removing package libcrypto1.0.0 from root...
24 Removing package libssl1.0.0 from root...
25 Removing package openssl from root...
26 Removing package openssl-conf from root...
...
31 Package opkg-collateral (1.0-r2) installed in root is up to date. 32 Package run-postinsts (1.0-r9) installed in root is up to date. 33 Package packagegroup-core-boot (1.0-r11) installed in root is up to date. 34 Package opkg (1:0.2.0-r0) installed in root is up to date.
35 Package poky-feed-config-opkg (1.0-r2) installed in root is up to date.
...

---------------------------------------------
Case 3: Incremental Rebuild
1. Edit local.conf, enable multilib and ipk incremental image generation
...
46 MACHINE ?= "qemux86-64"
254 require conf/multilib.conf
255 MULTILIBS = "multilib:lib32"
256 DEFAULTTUNE_virtclass-multilib-lib32 = "x86"
257
258 # IMAGE_INSTALL_append = " lib32-gzip openssl"
260 IMAGE_FEATURES_append = " package-management ssh-server-dropbear " 262
263 INC_IPK_IMAGE_GEN = "1"
...
bb core-image-minimal

2. Edit local.conf, append 'lib32-gzip openssl' to IMAGE_INSTALL vim local.conf
...
258 IMAGE_INSTALL_append = " lib32-gzip openssl"
...

3. bb core-image-minimal, lib32-gzip and openssl should be added.
vim log.do_rootfs:
...
20 NOTE: Package packagegroup-core-ssh-dropbear (1.0-r1) installed in root is up to date. 21 Package opkg-collateral (1.0-r2) installed in root is up to date.
22 Package run-postinsts (1.0-r9) installed in root is up to date.
23 Installing openssl (1.0.1e-r15.0) to root...
...
31 Package packagegroup-core-boot (1.0-r11) installed in root is up to date. 32 Package opkg (1:0.2.0-r0) installed in root is up to date.
33 Package poky-feed-config-opkg (1.0-r2) installed in root is up to date.
...
54 NOTE: Installing the following packages: lib32-gzip
...

---------------------------------------------
Case 4: Rebuild in place and Upgrade package
1. Edit local.conf, enable ipk incremental image generation
...
258 IMAGE_INSTALL_append = " gzip"
263 INC_IPK_IMAGE_GEN = "1"
...
bb core-image-minimal

2. Edit local.conf, upgrade gzip's PR
...
258 PR_pn-gzip = "r1"
...

3. bb core-image-minimal, gzip should be upgraded to r1
vim log.do_rootfs:
...
21 NOTE: Package packagegroup-core-ssh-dropbear (1.0-r1) installed in root is up to date. 22 Package opkg-collateral (1.0-r2) installed in root is up to date.
23 Package run-postinsts (1.0-r9) installed in root is up to date.
24 Package packagegroup-core-boot (1.0-r11) installed in root is up to date. 25 Package opkg (1:0.2.0-r0) installed in root is up to date.
26 Upgrading gzip on root from 1.6-r0 to 1.6-r1...
...

---------------------------------------------
Case 5: Test BAD_RECOMMENDATIONS
1. Edit local.conf, enable ipk incremental image generation
...
258 IMAGE_INSTALL_append = " util-linux"
263 INC_IPK_IMAGE_GEN = "1"
...
bb core-image-minimal

The util-linux-fdisk is in util-linux's RRECOMMENDS vim installed_pkgs.txt
...
41 util-linux-fdisk core2-64
...

2. Edit local.conf, add some spaces to IMAGE_INSTALL and add util-linux-fdisk to core-image-minimal's BAD_RECOMMENDATIONS
...
260 IMAGE_INSTALL_append = " util-linux "
265 BAD_RECOMMENDATIONS_pn-core-image-minimal += "util-linux-fdisk"
...

3. bb core-image-minimal, util-linux-fdisk should not be installed vim log.do_rootfs:
...
34 NOTE: Requested ignored recommendation util-linux-fdisk is not a package

    ...

119 util-linux: ignoring recommendation for util-linux-fdisk at user request 120 util-linux: unsatisfied recommendation for util-linux-fdisk
...

The util-linux-fdisk doesn't exist in installed_pkgs.txt any more.

---------------------------------------------
Case 5: Test NO_RECOMMENDATIONS
1. Edit local.conf, enable ipk incremental image generation
...
258 IMAGE_INSTALL_append = " util-linux"
263 INC_IPK_IMAGE_GEN = "1"
...
bb core-image-minimal

The util-linux-fdisk util-linux-cfdisk util-linux-sfdisk util-linux-mount util-linux-readprofile util-linux-mkfs are in util-linux's RRECOMMENDS

vim installed_pkgs.txt
...
4 util-linux-mkfs core2-64
18 util-linux-sfdisk core2-64
20 util-linux-mount core2-64
38 util-linux-cfdisk core2-64
41 util-linux-fdisk core2-64
46 util-linux-readprofile core2-64
...

2. Edit local.conf, add some spaces to IMAGE_INSTALL and assign core-image-minimal's NO_RECOMMENDATIONS with 1
...
260 IMAGE_INSTALL_append = " util-linux "
267 NO_RECOMMENDATIONS_pn-core-image-minimal = "1"
...

3. bb core-image-minimal, The six packages should not be installed vim log.do_rootfs:
...
34 NOTE: Requested ignored recommendation util-linux-fdisk is not a package

    ...

113 util-linux: ignoring recommendation for util-linux-sfdisk at user request 114 util-linux: unsatisfied recommendation for util-linux-sfdisk
115 util-linux: ignoring recommendation for util-linux-cfdisk at user request 116 util-linux: unsatisfied recommendation for util-linux-cfdisk
117 util-linux: ignoring recommendation for util-linux-mount at user request 118 util-linux: unsatisfied recommendation for util-linux-mount
119 util-linux: ignoring recommendation for util-linux-fdisk at user request 120 util-linux: unsatisfied recommendation for util-linux-fdisk
121 util-linux: ignoring recommendation for util-linux-mkfs at user request 122 util-linux: unsatisfied recommendation for util-linux-mkfs
123 util-linux: ignoring recommendation for util-linux-readprofile at user request 124 util-linux: unsatisfied recommendation for util-linux-readprofile
...

They don't exist in installed_pkgs.txt any more.

//Hongxu

The following changes since commit ac499f0b9f8a421a4e63d2de2a0b89f20c43d81c:

dev-manual: Fixed code block example for ARCHIVER. (2014-02-17 15:39:46 +0000)

are available in the git repository at:

git://git.pokylinux.org/poky-contrib hongxu/ipk-inc-gen
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=hongxu/ipk-inc-gen

Hongxu Jia (5):
package_manager.py: support ipk incremental image generation package_manager.py: tweak handle_bad_recommendations for ipk
incremental image generation
rootfs.py: support ipk incremental image generation rootfs.py: fix BAD_RECOMMENDATIONS for ipk incremental image
generation
rootfs.py: tweak _multilib_sanity_test for ipk incremental image
generation

 meta/lib/oe/package_manager.py | 124 +++++++++++++++++++++++++++++++++++---
 meta/lib/oe/rootfs.py          | 134 ++++++++++++++++++++++++++++++++++++++---
 2 files changed, 242 insertions(+), 16 deletions(-)
Signature · 1 line
1.8.1.2
author
#1 lore raw

PATCH 1/5 package_manager.py: support ipk incremental image generation

While incremental image generation enabled, 'load_old_install_solution' is used to determine what we've got in the previous (existed) image and 'dump_install_solution' to determine what we need to install in the current image.

The 'backup_packaging_data' is used to back up the current opkg database.

The 'recovery_packaging_data' is used to recover the opkg database which backed up by the previous image creation.

Tweak 'remove' function in OpkgPM class, which the options for remove with dependencies was incorrect.

[YOCTO #1894]

---
 meta/lib/oe/package_manager.py | 106 +++++++++++++++++++++++++++++++++++++++--
 1 file changed, 102 insertions(+), 4 deletions(-)
meta/lib/oe/package_manager.py +102 -4
index 6dc8fbd..4ea9677 100644
@@ -934,12 +934,13 @@ class RpmPM(PackageManager):
 
 
 class OpkgPM(PackageManager):
-    def __init__(self, d, target_rootfs, config_file, archs):
+    def __init__(self, d, target_rootfs, config_file, archs, task_name='target'):
         super(OpkgPM, self).__init__(d)
 
         self.target_rootfs = target_rootfs
         self.config_file = config_file
         self.pkg_archs = archs
+        self.task_name = task_name
 
         self.deploy_dir = self.d.getVar("DEPLOY_DIR_IPK", True)
         self.deploy_lock_file = os.path.join(self.deploy_dir, "deploy.lock")
@@ -956,6 +957,13 @@ class OpkgPM(PackageManager):
 
         bb.utils.mkdirhier(self.opkg_dir)
 
+        self.solution_manifest = self.d.expand('${T}/saved/%s_solution' %
+                                               self.task_name)
+        self.saved_opkg_dir = self.d.expand('${T}/saved/%s' % self.task_name)
+        if not os.path.exists(self.d.expand('${T}/saved')):
+            bb.utils.mkdirhier(self.d.expand('${T}/saved'))
+
+
         if (self.d.getVar('BUILD_IMAGES_FROM_FEEDS', True) or "") != "1":
             self._create_config()
         else:
@@ -1075,7 +1083,9 @@ class OpkgPM(PackageManager):
 
         try:
             bb.note("Installing the following packages: %s" % ' '.join(pkgs))
-            subprocess.check_output(cmd.split())
+            bb.note(cmd)
+            output = subprocess.check_output(cmd.split())
+            bb.note(output)
         except subprocess.CalledProcessError as e:
             (bb.fatal, bb.note)[attempt_only]("Unable to install packages. "
                                               "Command '%s' returned %d:\n%s" %
@@ -1083,14 +1093,16 @@ class OpkgPM(PackageManager):
 
     def remove(self, pkgs, with_dependencies=True):
         if with_dependencies:
-            cmd = "%s %s remove %s" % \
+            cmd = "%s %s --force-depends --force-remove --force-removal-of-dependent-packages remove %s" % \
                 (self.opkg_cmd, self.opkg_args, ' '.join(pkgs))
         else:
             cmd = "%s %s --force-depends remove %s" % \
                 (self.opkg_cmd, self.opkg_args, ' '.join(pkgs))
 
         try:
-            subprocess.check_output(cmd.split())
+            bb.note(cmd)
+            output = subprocess.check_output(cmd.split())
+            bb.note(output)
         except subprocess.CalledProcessError as e:
             bb.fatal("Unable to remove packages. Command '%s' "
                      "returned %d:\n%s" % (e.cmd, e.returncode, e.output))
@@ -1175,6 +1187,92 @@ class OpkgPM(PackageManager):
                     else:
                         status.write(line + "\n")
 
+    '''
+    If incremental install, we need to determine what we've got,
+    what we need to add, and what to remove...
+    The dump_install_solution will dump and save the new install
+    solution.
+    '''
+    def dump_install_solution(self, pkgs):
+        bb.note('creating new install solution for incremental install')
+        if len(pkgs) == 0:
+            return
+
+        install_pkgs = list()
+
+        # Create an temp dir as opkg root for simulating the installation
+        temp_rootfs = self.d.expand('${T}/opkg')
+        temp_opkg_dir = os.path.join(temp_rootfs, 'var/lib/opkg')
+        bb.utils.mkdirhier(temp_opkg_dir)
+
+        opkg_args = "-f %s -o %s " % (self.config_file, temp_rootfs)
+        opkg_args += self.d.getVar("OPKG_ARGS", True)
+
+        cmd = "%s %s update" % (self.opkg_cmd,
+                                opkg_args)
+        try:
+            subprocess.check_output(cmd, shell=True)
+        except subprocess.CalledProcessError as e:
+            bb.note("Unable to dump install packages. Command '%s' "
+                    "returned %d:\n%s" % (cmd, e.returncode, e.output))
+
+        # Simulate installation from zero
+        cmd = "%s %s --noaction install %s " % (self.opkg_cmd,
+                                                opkg_args,
+                                                ' '.join(pkgs))
+        cmd += "| awk '/^Installing/{print $2}' "
+        cmd += "| sort -u -o %s" % self.solution_manifest
+        try:
+            subprocess.check_output(cmd, shell=True)
+            with open(self.solution_manifest, 'r') as manifest:
+                for pkg in manifest.read().split('\n'):
+                    install_pkgs.append(pkg)
+        except subprocess.CalledProcessError as e:
+            bb.note("Unable to dump install packages. Command '%s' "
+                    "returned %d:\n%s" % (cmd, e.returncode, e.output))
+
+        bb.utils.remove(temp_rootfs, True)
+
+        return install_pkgs
+
+    '''
+    If incremental install, we need to determine what we've got,
+    what we need to add, and what to remove...
+    The load_old_install_solution will load the previous install
+    solution
+    '''
+    def load_old_install_solution(self):
+        bb.note('load old install solution for incremental install')
+        installed_pkgs = list()
+        if not os.path.exists(self.solution_manifest):
+            bb.note('old install solution not exist')
+            return installed_pkgs
+
+        with open(self.solution_manifest, 'r') as manifest:
+            for pkg in manifest.read().split('\n'):
+                installed_pkgs.append(pkg.strip())
+
+        return installed_pkgs
+
+    def backup_packaging_data(self):
+        # Save the opkglib for increment rpm image generation
+        if os.path.exists(self.saved_opkg_dir):
+            bb.utils.remove(self.saved_opkg_dir, True)
+        shutil.copytree(self.opkg_dir,
+                        self.saved_opkg_dir,
+                        symlinks=True)
+
+    def recovery_packaging_data(self):
+        # Move the opkglib back
+        if os.path.exists(self.saved_opkg_dir):
+            if os.path.exists(self.opkg_dir):
+                bb.utils.remove(self.opkg_dir, True)
+
+            bb.note('Recovery packaging data')
+            shutil.copytree(self.saved_opkg_dir,
+                            self.opkg_dir,
+                            symlinks=True)
+
 
 class DpkgPM(PackageManager):
     def __init__(self, d, target_rootfs, archs, base_archs, apt_conf_dir=None):
Signature · 1 line
1.8.1.2
#2 lore raw

Re: [PATCH 1/5] package_manager.py: support ipk incremental image generation

Quoted 84 lines · Hongxu Jia

While incremental image generation enabled, 'load_old_install_solution' is used to determine what we've got in the previous (existed) image and 'dump_install_solution' to determine what we need to install in the current image.

The 'backup_packaging_data' is used to back up the current opkg database.

The 'recovery_packaging_data' is used to recover the opkg database which backed up by the previous image creation.

Tweak 'remove' function in OpkgPM class, which the options for remove with dependencies was incorrect.

[YOCTO #1894]

---
 meta/lib/oe/package_manager.py | 106 +++++++++++++++++++++++++++++++++++++++--
 1 file changed, 102 insertions(+), 4 deletions(-)
meta/lib/oe/package_manager.py +16 -4
index 6dc8fbd..4ea9677 100644
@@ -934,12 +934,13 @@ class RpmPM(PackageManager):
 
 
 class OpkgPM(PackageManager):
-    def __init__(self, d, target_rootfs, config_file, archs):
+    def __init__(self, d, target_rootfs, config_file, archs, task_name='target'):
         super(OpkgPM, self).__init__(d)
 
         self.target_rootfs = target_rootfs
         self.config_file = config_file
         self.pkg_archs = archs
+        self.task_name = task_name
 
         self.deploy_dir = self.d.getVar("DEPLOY_DIR_IPK", True)
         self.deploy_lock_file = os.path.join(self.deploy_dir, "deploy.lock")
@@ -956,6 +957,13 @@ class OpkgPM(PackageManager):
 
         bb.utils.mkdirhier(self.opkg_dir)
 
+        self.solution_manifest = self.d.expand('${T}/saved/%s_solution' %
+                                               self.task_name)
+        self.saved_opkg_dir = self.d.expand('${T}/saved/%s' % self.task_name)
+        if not os.path.exists(self.d.expand('${T}/saved')):
+            bb.utils.mkdirhier(self.d.expand('${T}/saved'))
+
+
         if (self.d.getVar('BUILD_IMAGES_FROM_FEEDS', True) or "") != "1":
             self._create_config()
         else:
@@ -1075,7 +1083,9 @@ class OpkgPM(PackageManager):
 
         try:
             bb.note("Installing the following packages: %s" % ' '.join(pkgs))
-            subprocess.check_output(cmd.split())
+            bb.note(cmd)
+            output = subprocess.check_output(cmd.split())
+            bb.note(output)
         except subprocess.CalledProcessError as e:
             (bb.fatal, bb.note)[attempt_only]("Unable to install packages. "
                                               "Command '%s' returned %d:\n%s" %
@@ -1083,14 +1093,16 @@ class OpkgPM(PackageManager):
 
     def remove(self, pkgs, with_dependencies=True):
         if with_dependencies:
-            cmd = "%s %s remove %s" % \
+            cmd = "%s %s --force-depends --force-remove --force-removal-of-dependent-packages remove %s" % \
                 (self.opkg_cmd, self.opkg_args, ' '.join(pkgs))
         else:
             cmd = "%s %s --force-depends remove %s" % \
                 (self.opkg_cmd, self.opkg_args, ' '.join(pkgs))
 
         try:
-            subprocess.check_output(cmd.split())
+            bb.note(cmd)
+            output = subprocess.check_output(cmd.split())
+            bb.note(output)
         except subprocess.CalledProcessError as e:
             bb.fatal("Unable to remove packages. Command '%s' "
                      "returned %d:\n%s" % (e.cmd, e.returncode, e.output))

meta/lib/oe/package_manager.py class OpkgPM(PackageManager): · line 1196

Quoted 10 lines
meta/lib/oe/package_manager.py +7 -0
@@ -1175,6 +1187,92 @@ class OpkgPM(PackageManager):
                     else:
                         status.write(line + "\n")
 
+    '''
+    If incremental install, we need to determine what we've got,
+    what we need to add, and what to remove...
+    The dump_install_solution will dump and save the new install
+    solution.
+    '''
+    def dump_install_solution(self, pkgs):

Why not have this function in the Manifest class? We have an API in
place: Manifest.create_final() that should probably take care of anything related to manifest(s) creation after all packages are installed.

Quoted 18 lines
meta/lib/oe/package_manager.py +18 -0
+        bb.note('creating new install solution for incremental install')
+        if len(pkgs) == 0:
+            return
+
+        install_pkgs = list()
+
+        # Create an temp dir as opkg root for simulating the installation
+        temp_rootfs = self.d.expand('${T}/opkg')
+        temp_opkg_dir = os.path.join(temp_rootfs, 'var/lib/opkg')
+        bb.utils.mkdirhier(temp_opkg_dir)
+
+        opkg_args = "-f %s -o %s " % (self.config_file, temp_rootfs)
+        opkg_args += self.d.getVar("OPKG_ARGS", True)
+
+        cmd = "%s %s update" % (self.opkg_cmd,
+                                opkg_args)
+        try:
+            subprocess.check_output(cmd, shell=True)

meta/lib/oe/package_manager.py class OpkgPM(PackageManager): · line 1224

Quoted 10 lines
meta/lib/oe/package_manager.py +10 -0
+        except subprocess.CalledProcessError as e:
+            bb.note("Unable to dump install packages. Command '%s' "
+                    "returned %d:\n%s" % (cmd, e.returncode, e.output))
+
+        # Simulate installation from zero
+        cmd = "%s %s --noaction install %s " % (self.opkg_cmd,
+                                                opkg_args,
+                                                ' '.join(pkgs))
+        cmd += "| awk '/^Installing/{print $2}' "
+        cmd += "| sort -u -o %s" % self.solution_manifest

Do we need to preserve this bash one-liner? Why not handle the command output using Python?

Quoted 10 lines
meta/lib/oe/package_manager.py +10 -0
+        try:
+            subprocess.check_output(cmd, shell=True)
+            with open(self.solution_manifest, 'r') as manifest:
+                for pkg in manifest.read().split('\n'):
+                    install_pkgs.append(pkg)
+        except subprocess.CalledProcessError as e:
+            bb.note("Unable to dump install packages. Command '%s' "
+                    "returned %d:\n%s" % (cmd, e.returncode, e.output))
+
+        bb.utils.remove(temp_rootfs, True)

meta/lib/oe/package_manager.py class OpkgPM(PackageManager): · line 1244

Quoted 10 lines
meta/lib/oe/package_manager.py +10 -0
+
+        return install_pkgs
+
+    '''
+    If incremental install, we need to determine what we've got,
+    what we need to add, and what to remove...
+    The load_old_install_solution will load the previous install
+    solution
+    '''
+    def load_old_install_solution(self):

Same here: why not put it in the Manifest class? As we already have the Manifest.parse_initial_manifest(), we can have something similar for the final_manifest (or install_solution, etc) parsing.

Quoted 4 lines
meta/lib/oe/package_manager.py +4 -0
+        bb.note('load old install solution for incremental install')
+        installed_pkgs = list()
+        if not os.path.exists(self.solution_manifest):
+            bb.note('old install solution not exist')

meta/lib/oe/package_manager.py class OpkgPM(PackageManager): · line 1258

Quoted 10 lines
meta/lib/oe/package_manager.py +10 -0
+            return installed_pkgs
+
+        with open(self.solution_manifest, 'r') as manifest:
+            for pkg in manifest.read().split('\n'):
+                installed_pkgs.append(pkg.strip())
+
+        return installed_pkgs
+
+    def backup_packaging_data(self):
+        # Save the opkglib for increment rpm image generation

s/rpm/opkg/

meta/lib/oe/package_manager.py class OpkgPM(PackageManager): · line 1265

Quoted 7 lines
meta/lib/oe/package_manager.py +7 -0
+        if os.path.exists(self.saved_opkg_dir):
+            bb.utils.remove(self.saved_opkg_dir, True)
+        shutil.copytree(self.opkg_dir,
+                        self.saved_opkg_dir,
+                        symlinks=True)
+
+    def recovery_packaging_data(self):

s/recovery/recover/ ?

Quoted 16 lines
meta/lib/oe/package_manager.py +10 -0
+        # Move the opkglib back
+        if os.path.exists(self.saved_opkg_dir):
+            if os.path.exists(self.opkg_dir):
+                bb.utils.remove(self.opkg_dir, True)
+
+            bb.note('Recovery packaging data')
+            shutil.copytree(self.saved_opkg_dir,
+                            self.opkg_dir,
+                            symlinks=True)
+
 
 class DpkgPM(PackageManager):
     def __init__(self, d, target_rootfs, archs, base_archs, apt_conf_dir=None):
Signature · 1 line
1.8.1.2
author
#3 lore raw

Re: [PATCH 1/5] package_manager.py: support ipk incremental image generation

Quoted 88 lines · Laurentiu Palcu
Quoted 87 lines · Hongxu Jia

While incremental image generation enabled, 'load_old_install_solution' is used to determine what we've got in the previous (existed) image and 'dump_install_solution' to determine what we need to install in the current image.

The 'backup_packaging_data' is used to back up the current opkg database.

The 'recovery_packaging_data' is used to recover the opkg database which backed up by the previous image creation.

Tweak 'remove' function in OpkgPM class, which the options for remove with dependencies was incorrect.

[YOCTO #1894]

---
  meta/lib/oe/package_manager.py | 106 +++++++++++++++++++++++++++++++++++++++--
  1 file changed, 102 insertions(+), 4 deletions(-)
meta/lib/oe/package_manager.py +16 -4
index 6dc8fbd..4ea9677 100644
@@ -934,12 +934,13 @@ class RpmPM(PackageManager):
 
 
  class OpkgPM(PackageManager):
-    def __init__(self, d, target_rootfs, config_file, archs):
+    def __init__(self, d, target_rootfs, config_file, archs, task_name='target'):
          super(OpkgPM, self).__init__(d)
 
          self.target_rootfs = target_rootfs
          self.config_file = config_file
          self.pkg_archs = archs
+        self.task_name = task_name
 
          self.deploy_dir = self.d.getVar("DEPLOY_DIR_IPK", True)
          self.deploy_lock_file = os.path.join(self.deploy_dir, "deploy.lock")
@@ -956,6 +957,13 @@ class OpkgPM(PackageManager):
 
          bb.utils.mkdirhier(self.opkg_dir)
 
+        self.solution_manifest = self.d.expand('${T}/saved/%s_solution' %
+                                               self.task_name)
+        self.saved_opkg_dir = self.d.expand('${T}/saved/%s' % self.task_name)
+        if not os.path.exists(self.d.expand('${T}/saved')):
+            bb.utils.mkdirhier(self.d.expand('${T}/saved'))
+
+
          if (self.d.getVar('BUILD_IMAGES_FROM_FEEDS', True) or "") != "1":
              self._create_config()
          else:
@@ -1075,7 +1083,9 @@ class OpkgPM(PackageManager):
 
          try:
              bb.note("Installing the following packages: %s" % ' '.join(pkgs))
-            subprocess.check_output(cmd.split())
+            bb.note(cmd)
+            output = subprocess.check_output(cmd.split())
+            bb.note(output)
          except subprocess.CalledProcessError as e:
              (bb.fatal, bb.note)[attempt_only]("Unable to install packages. "
                                                "Command '%s' returned %d:\n%s" %
@@ -1083,14 +1093,16 @@ class OpkgPM(PackageManager):
 
      def remove(self, pkgs, with_dependencies=True):
          if with_dependencies:
-            cmd = "%s %s remove %s" % \
+            cmd = "%s %s --force-depends --force-remove --force-removal-of-dependent-packages remove %s" % \
                  (self.opkg_cmd, self.opkg_args, ' '.join(pkgs))
          else:
              cmd = "%s %s --force-depends remove %s" % \
                  (self.opkg_cmd, self.opkg_args, ' '.join(pkgs))
 
          try:
-            subprocess.check_output(cmd.split())
+            bb.note(cmd)
+            output = subprocess.check_output(cmd.split())
+            bb.note(output)
          except subprocess.CalledProcessError as e:
              bb.fatal("Unable to remove packages. Command '%s' "
                       "returned %d:\n%s" % (e.cmd, e.returncode, e.output))
@@ -1175,6 +1187,92 @@ class OpkgPM(PackageManager):
                      else:
                          status.write(line + "\n")
 
Quoted 10 lines

meta/lib/oe/package_manager.py class OpkgPM(PackageManager): · line 1196

Quoted 7 lines
meta/lib/oe/package_manager.py +7 -0
+    '''
+    If incremental install, we need to determine what we've got,
+    what we need to add, and what to remove...
+    The dump_install_solution will dump and save the new install
+    solution.
+    '''
+    def dump_install_solution(self, pkgs):

Why not have this function in the Manifest class? We have an API in
place: Manifest.create_final() that should probably take care of anything related to manifest(s) creation after all packages are installed.

Got it,

The dump_install_solution creates a manifest with the packages that *will be* installed into the image, it is a dummy installation.

I will move 'dump_install_solution' to Manifest class as create_full.

Quoted 22 lines
Quoted 22 lines
meta/lib/oe/package_manager.py +22 -0
+        bb.note('creating new install solution for incremental install')
+        if len(pkgs) == 0:
+            return
+
+        install_pkgs = list()
+
+        # Create an temp dir as opkg root for simulating the installation
+        temp_rootfs = self.d.expand('${T}/opkg')
+        temp_opkg_dir = os.path.join(temp_rootfs, 'var/lib/opkg')
+        bb.utils.mkdirhier(temp_opkg_dir)
+
+        opkg_args = "-f %s -o %s " % (self.config_file, temp_rootfs)
+        opkg_args += self.d.getVar("OPKG_ARGS", True)
+
+        cmd = "%s %s update" % (self.opkg_cmd,
+                                opkg_args)
+        try:
+            subprocess.check_output(cmd, shell=True)
+        except subprocess.CalledProcessError as e:
+            bb.note("Unable to dump install packages. Command '%s' "
+                    "returned %d:\n%s" % (cmd, e.returncode, e.output))
+
Quoted 8 lines

meta/lib/oe/package_manager.py class OpkgPM(PackageManager): · line 1224

Quoted 6 lines
meta/lib/oe/package_manager.py +6 -0
+        # Simulate installation from zero
+        cmd = "%s %s --noaction install %s " % (self.opkg_cmd,
+                                                opkg_args,
+                                                ' '.join(pkgs))
+        cmd += "| awk '/^Installing/{print $2}' "
+        cmd += "| sort -u -o %s" % self.solution_manifest

Do we need to preserve this bash one-liner? Why not handle the command output using Python?

Agree, I will handle this in Python.

Quoted 15 lines
Quoted 15 lines
meta/lib/oe/package_manager.py +15 -0
+        try:
+            subprocess.check_output(cmd, shell=True)
+            with open(self.solution_manifest, 'r') as manifest:
+                for pkg in manifest.read().split('\n'):
+                    install_pkgs.append(pkg)
+        except subprocess.CalledProcessError as e:
+            bb.note("Unable to dump install packages. Command '%s' "
+                    "returned %d:\n%s" % (cmd, e.returncode, e.output))
+
+        bb.utils.remove(temp_rootfs, True)
+
+        return install_pkgs
+
+    '''
+    If incremental install, we need to determine what we've got,
Quoted 8 lines

meta/lib/oe/package_manager.py class OpkgPM(PackageManager): · line 1244

Quoted 5 lines
meta/lib/oe/package_manager.py +5 -0
+    what we need to add, and what to remove...
+    The load_old_install_solution will load the previous install
+    solution
+    '''
+    def load_old_install_solution(self):

Same here: why not put it in the Manifest class? As we already have the Manifest.parse_initial_manifest(), we can have something similar for the final_manifest (or install_solution, etc) parsing.

Agree, Manifest.parse_full_manifest() will be added

Quoted 7 lines
Quoted 7 lines
meta/lib/oe/package_manager.py +7 -0
+        bb.note('load old install solution for incremental install')
+        installed_pkgs = list()
+        if not os.path.exists(self.solution_manifest):
+            bb.note('old install solution not exist')
+            return installed_pkgs
+
+        with open(self.solution_manifest, 'r') as manifest:
Quoted 8 lines

meta/lib/oe/package_manager.py class OpkgPM(PackageManager): · line 1258

Quoted 7 lines
meta/lib/oe/package_manager.py +7 -0
+            for pkg in manifest.read().split('\n'):
+                installed_pkgs.append(pkg.strip())
+
+        return installed_pkgs
+
+    def backup_packaging_data(self):
+        # Save the opkglib for increment rpm image generation

s/rpm/opkg/

Sorry for the typo

Quoted 8 lines

meta/lib/oe/package_manager.py class OpkgPM(PackageManager): · line 1265

Quoted 7 lines
meta/lib/oe/package_manager.py +7 -0
+        if os.path.exists(self.saved_opkg_dir):
+            bb.utils.remove(self.saved_opkg_dir, True)
+        shutil.copytree(self.opkg_dir,
+                        self.saved_opkg_dir,
+                        symlinks=True)
+
+    def recovery_packaging_data(self):

s/recovery/recover/ ?

Sorry for the typo

V2 incoming

//Hongxu

Quoted 16 lines
Quoted 16 lines
meta/lib/oe/package_manager.py +10 -0
+        # Move the opkglib back
+        if os.path.exists(self.saved_opkg_dir):
+            if os.path.exists(self.opkg_dir):
+                bb.utils.remove(self.opkg_dir, True)
+
+            bb.note('Recovery packaging data')
+            shutil.copytree(self.saved_opkg_dir,
+                            self.opkg_dir,
+                            symlinks=True)
+
 
  class DpkgPM(PackageManager):
      def __init__(self, d, target_rootfs, archs, base_archs, apt_conf_dir=None):
Signature · 1 line
1.8.1.2
author
#4 lore raw

PATCH 2/5 package_manager.py: tweak handle_bad_recommendations for ipk incremental image generation

In reply to Hongxu Jia

Let opkg_dir be configurable in handle_bad_recommendations, so it could work on other root fs.

Invoke handle_bad_recommendations in dump_install_solution which simulates package install on other root fs.

[YOCTO #1894]
---
 meta/lib/oe/package_manager.py | 18 +++++++++++++-----
 1 file changed, 13 insertions(+), 5 deletions(-)
meta/lib/oe/package_manager.py +13 -5
index 4ea9677..3cfd4af 100644
@@ -1157,12 +1157,19 @@ class OpkgPM(PackageManager):
 
         return output
 
-    def handle_bad_recommendations(self):
-        bad_recommendations = self.d.getVar("BAD_RECOMMENDATIONS", True)
-        if bad_recommendations is None:
+    def handle_bad_recommendations(self, opkg_dir=""):
+        bad_recommendations = self.d.getVar("BAD_RECOMMENDATIONS", True) or ""
+        if bad_recommendations.strip() == "":
             return
 
-        status_file = os.path.join(self.opkg_dir, "status")
+        if opkg_dir == "":
+            opkg_dir = self.opkg_dir
+        status_file = os.path.join(opkg_dir, "status")
+
+        # If status file existed, it means the bad recommendations has already
+        # been handled
+        if os.path.exists(status_file):
+            return
 
         cmd = "%s %s info " % (self.opkg_cmd, self.opkg_args)
 
@@ -1177,7 +1184,7 @@ class OpkgPM(PackageManager):
                              "returned %d:\n%s" % (pkg_info, e.returncode, e.output))
 
                 if output == "":
-                    bb.note("Requested ignored recommendation $i is "
+                    bb.note("Requested ignored recommendation %s is "
                             "not a package" % pkg)
                     continue
 
@@ -1204,6 +1211,7 @@ class OpkgPM(PackageManager):
         temp_rootfs = self.d.expand('${T}/opkg')
         temp_opkg_dir = os.path.join(temp_rootfs, 'var/lib/opkg')
         bb.utils.mkdirhier(temp_opkg_dir)
+        self.handle_bad_recommendations(temp_opkg_dir)
 
         opkg_args = "-f %s -o %s " % (self.config_file, temp_rootfs)
         opkg_args += self.d.getVar("OPKG_ARGS", True)
Signature · 1 line
1.8.1.2
author
#5 lore raw

PATCH 3/5 rootfs.py: support ipk incremental image generation

In reply to Hongxu Jia

While incremental image generation enabled, if previous image existed, it restores the opkg database. Based on the previous image, it gets what need to remove for the current and remove them.

The newly added and upgraded packages will be done since opkg install invoked.

[YOCTO #1894]

---
 meta/lib/oe/rootfs.py | 50 +++++++++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 47 insertions(+), 3 deletions(-)
meta/lib/oe/rootfs.py +47 -3
index d149ca3..5561bb9 100644
@@ -438,13 +438,25 @@ class OpkgRootfs(Rootfs):
     def __init__(self, d, manifest_dir):
         super(OpkgRootfs, self).__init__(d)
 
-        bb.utils.remove(self.image_rootfs, True)
-        bb.utils.remove(self.d.getVar('MULTILIB_TEMP_ROOTFS', True), True)
         self.manifest = OpkgManifest(d, manifest_dir)
         self.opkg_conf = self.d.getVar("IPKGCONF_TARGET", True)
         self.pkg_archs = self.d.getVar("ALL_MULTILIB_PACKAGE_ARCHS", True)
 
-        self.pm = OpkgPM(d, self.image_rootfs, self.opkg_conf, self.pkg_archs)
+        self.inc_opkg_image_gen = self.d.getVar('INC_IPK_IMAGE_GEN', True)
+        if self.inc_opkg_image_gen != "1":
+            bb.utils.remove(self.image_rootfs, True)
+            self.pm = OpkgPM(d,
+                             self.image_rootfs,
+                             self.opkg_conf,
+                             self.pkg_archs)
+        else:
+            self.pm = OpkgPM(d,
+                             self.image_rootfs,
+                             self.opkg_conf,
+                             self.pkg_archs)
+            self.pm.recovery_packaging_data()
+
+        bb.utils.remove(self.d.getVar('MULTILIB_TEMP_ROOTFS', True), True)
 
     """
     This function was reused from the old implementation.
@@ -508,6 +520,32 @@ class OpkgRootfs(Rootfs):
 
         self._multilib_sanity_test(dirs)
 
+    '''
+    While ipk incremental image generation is enabled, it will remove the
+    unneeded pkgs by comparing the new install solution manifest and the
+    old installed manifest.
+    '''
+    def _create_incremental(self, pkgs_initial_install):
+        if self.inc_opkg_image_gen == "1":
+
+            pkgs_to_install = list()
+            for pkg_type in pkgs_initial_install:
+                pkgs_to_install += pkgs_initial_install[pkg_type]
+
+            installed_manifest = self.pm.load_old_install_solution()
+            solution_manifest = self.pm.dump_install_solution(pkgs_to_install)
+
+            pkg_to_remove = list()
+            for pkg in installed_manifest:
+                if pkg not in solution_manifest:
+                    pkg_to_remove.append(pkg)
+
+            self.pm.update()
+
+            if pkg_to_remove != []:
+                bb.note('incremental removed: %s' % ' '.join(pkg_to_remove))
+                self.pm.remove(pkg_to_remove)
+
     def _create(self):
         pkgs_to_install = self.manifest.parse_initial_manifest()
         opkg_pre_process_cmds = self.d.getVar('OPKG_PREPROCESS_COMMANDS', True)
@@ -518,6 +556,9 @@ class OpkgRootfs(Rootfs):
         if (self.d.getVar('BUILD_IMAGES_FROM_FEEDS', True) or "") != "1":
             self.pm.write_index()
 
+        if self.inc_opkg_image_gen == "1":
+            self._create_incremental(pkgs_to_install)
+
         execute_pre_post_process(self.d, opkg_pre_process_cmds)
 
         self.pm.update()
@@ -540,6 +581,9 @@ class OpkgRootfs(Rootfs):
         execute_pre_post_process(self.d, opkg_post_process_cmds)
         execute_pre_post_process(self.d, rootfs_post_install_cmds)
 
+        if self.inc_opkg_image_gen == "1":
+            self.pm.backup_packaging_data()
+
     def _get_delayed_postinsts(self):
         pkg_list = []
         status_file = os.path.join(self.image_rootfs,
Signature · 1 line
1.8.1.2
author
#6 lore raw

PATCH 4/5 rootfs.py: fix BAD_RECOMMENDATIONS for ipk incremental image generation

In reply to Hongxu Jia

While incremental image generation enabled and the previous image existed, if BAD_RECOMMENDATIONS is changed, the operation on the existed image is complicated, so remove the existed image in this situation.

The same with PACKAGE_EXCLUDE and NO_RECOMMENDATIONS.

[YOCTO #1894]
---
 meta/lib/oe/rootfs.py | 30 +++++++++++++++++++++++++++++-
 1 file changed, 29 insertions(+), 1 deletion(-)
meta/lib/oe/rootfs.py +29 -1
index 5561bb9..3d7adf9 100644
@@ -443,7 +443,7 @@ class OpkgRootfs(Rootfs):
         self.pkg_archs = self.d.getVar("ALL_MULTILIB_PACKAGE_ARCHS", True)
 
         self.inc_opkg_image_gen = self.d.getVar('INC_IPK_IMAGE_GEN', True)
-        if self.inc_opkg_image_gen != "1":
+        if self._remove_existed_image():
             bb.utils.remove(self.image_rootfs, True)
             self.pm = OpkgPM(d,
                              self.image_rootfs,
@@ -546,6 +546,34 @@ class OpkgRootfs(Rootfs):
                 bb.note('incremental removed: %s' % ' '.join(pkg_to_remove))
                 self.pm.remove(pkg_to_remove)
 
+    '''
+    Compare with previous (existed) image creation, if some conditions
+    triggered, the previous (existed) image should be removed.
+    The conditions included any of 'PACKAGE_EXCLUDE, NO_RECOMMENDATIONS
+    and BAD_RECOMMENDATIONS' has been changed.
+    '''
+    def _remove_existed_image(self):
+        # Incremental image creation is not enable
+        if self.inc_opkg_image_gen != "1":
+            return True
+
+        vars_list_dir = self.d.expand('${T}/vars_list')
+
+        old_vars_list = ""
+        if os.path.exists(vars_list_dir):
+            old_vars_list = open(vars_list_dir, 'r+').read()
+
+        new_vars_list = '%s:%s:%s\n' % \
+                ((self.d.getVar('BAD_RECOMMENDATIONS', True) or '').strip(),
+                 (self.d.getVar('NO_RECOMMENDATIONS', True) or '').strip(),
+                 (self.d.getVar('PACKAGE_EXCLUDE', True) or '').strip())
+        open(vars_list_dir, 'w+').write(new_vars_list)
+
+        if old_vars_list != new_vars_list:
+            return True
+
+        return False
+
     def _create(self):
         pkgs_to_install = self.manifest.parse_initial_manifest()
         opkg_pre_process_cmds = self.d.getVar('OPKG_PREPROCESS_COMMANDS', True)
Signature · 1 line
1.8.1.2
#7 lore raw

Re: [PATCH 4/5] rootfs.py: fix BAD_RECOMMENDATIONS for ipk incremental image generation

Quoted 26 lines · Hongxu Jia

While incremental image generation enabled and the previous image existed, if BAD_RECOMMENDATIONS is changed, the operation on the existed image is complicated, so remove the existed image in this situation.

The same with PACKAGE_EXCLUDE and NO_RECOMMENDATIONS.

[YOCTO #1894]
---
 meta/lib/oe/rootfs.py | 30 +++++++++++++++++++++++++++++-
 1 file changed, 29 insertions(+), 1 deletion(-)
meta/lib/oe/rootfs.py +1 -1
index 5561bb9..3d7adf9 100644
@@ -443,7 +443,7 @@ class OpkgRootfs(Rootfs):
         self.pkg_archs = self.d.getVar("ALL_MULTILIB_PACKAGE_ARCHS", True)
 
         self.inc_opkg_image_gen = self.d.getVar('INC_IPK_IMAGE_GEN', True)
-        if self.inc_opkg_image_gen != "1":
+        if self._remove_existed_image():
             bb.utils.remove(self.image_rootfs, True)
             self.pm = OpkgPM(d,
                              self.image_rootfs,

meta/lib/oe/rootfs.py class OpkgRootfs(Rootfs): · line 555

Quoted 10 lines
meta/lib/oe/rootfs.py +7 -0
@@ -546,6 +546,34 @@ class OpkgRootfs(Rootfs):
                 bb.note('incremental removed: %s' % ' '.join(pkg_to_remove))
                 self.pm.remove(pkg_to_remove)
 
+    '''
+    Compare with previous (existed) image creation, if some conditions
+    triggered, the previous (existed) image should be removed.
+    The conditions included any of 'PACKAGE_EXCLUDE, NO_RECOMMENDATIONS
+    and BAD_RECOMMENDATIONS' has been changed.
+    '''
+    def _remove_existed_image(self):

s/existed/existing/

Also in the comment above.

meta/lib/oe/rootfs.py class OpkgRootfs(Rootfs): · line 560

Quoted 5 lines
meta/lib/oe/rootfs.py +5 -0
+        # Incremental image creation is not enable
+        if self.inc_opkg_image_gen != "1":
+            return True
+
+        vars_list_dir = self.d.expand('${T}/vars_list')

s/vars_list_dir/vars_list_file/ in the line above and the code below. Leaving this variable with a 'dir' suffix while we're opening a file for reading/writing, is a little misleading.

Quoted 22 lines
meta/lib/oe/rootfs.py +16 -0
+
+        old_vars_list = ""
+        if os.path.exists(vars_list_dir):
+            old_vars_list = open(vars_list_dir, 'r+').read()
+
+        new_vars_list = '%s:%s:%s\n' % \
+                ((self.d.getVar('BAD_RECOMMENDATIONS', True) or '').strip(),
+                 (self.d.getVar('NO_RECOMMENDATIONS', True) or '').strip(),
+                 (self.d.getVar('PACKAGE_EXCLUDE', True) or '').strip())
+        open(vars_list_dir, 'w+').write(new_vars_list)
+
+        if old_vars_list != new_vars_list:
+            return True
+
+        return False
+
     def _create(self):
         pkgs_to_install = self.manifest.parse_initial_manifest()
         opkg_pre_process_cmds = self.d.getVar('OPKG_PREPROCESS_COMMANDS', True)
Signature · 1 line
1.8.1.2
author
#8 lore raw

Re: [PATCH 4/5] rootfs.py: fix BAD_RECOMMENDATIONS for ipk incremental image generation

Quoted 39 lines · Laurentiu Palcu
Quoted 26 lines · Hongxu Jia

While incremental image generation enabled and the previous image existed, if BAD_RECOMMENDATIONS is changed, the operation on the existed image is complicated, so remove the existed image in this situation.

The same with PACKAGE_EXCLUDE and NO_RECOMMENDATIONS.

[YOCTO #1894]
---
  meta/lib/oe/rootfs.py | 30 +++++++++++++++++++++++++++++-
  1 file changed, 29 insertions(+), 1 deletion(-)
meta/lib/oe/rootfs.py +1 -1
index 5561bb9..3d7adf9 100644
@@ -443,7 +443,7 @@ class OpkgRootfs(Rootfs):
          self.pkg_archs = self.d.getVar("ALL_MULTILIB_PACKAGE_ARCHS", True)
 
          self.inc_opkg_image_gen = self.d.getVar('INC_IPK_IMAGE_GEN', True)
-        if self.inc_opkg_image_gen != "1":
+        if self._remove_existed_image():
              bb.utils.remove(self.image_rootfs, True)
              self.pm = OpkgPM(d,
                               self.image_rootfs,

meta/lib/oe/rootfs.py class OpkgRootfs(Rootfs): · line 555

Quoted 10 lines
meta/lib/oe/rootfs.py +7 -0
@@ -546,6 +546,34 @@ class OpkgRootfs(Rootfs):
                  bb.note('incremental removed: %s' % ' '.join(pkg_to_remove))
                  self.pm.remove(pkg_to_remove)
 
+    '''
+    Compare with previous (existed) image creation, if some conditions
+    triggered, the previous (existed) image should be removed.
+    The conditions included any of 'PACKAGE_EXCLUDE, NO_RECOMMENDATIONS
+    and BAD_RECOMMENDATIONS' has been changed.
+    '''
+    def _remove_existed_image(self):

s/existed/existing/

Quoted 1 line

Also in the comment above.

Sorry for the typo

Quoted 9 lines

meta/lib/oe/rootfs.py class OpkgRootfs(Rootfs): · line 560

Quoted 6 lines
meta/lib/oe/rootfs.py +5 -0
+        # Incremental image creation is not enable
+        if self.inc_opkg_image_gen != "1":
+            return True
+
+        vars_list_dir = self.d.expand('${T}/vars_list')

s/vars_list_dir/vars_list_file/ in the line above and the code below. Leaving this variable with a 'dir' suffix while we're opening a file for reading/writing, is a little misleading.

Sorry for the misleading

V2 incoming

//Hongxu

Quoted 22 lines
Quoted 22 lines
meta/lib/oe/rootfs.py +16 -0
+
+        old_vars_list = ""
+        if os.path.exists(vars_list_dir):
+            old_vars_list = open(vars_list_dir, 'r+').read()
+
+        new_vars_list = '%s:%s:%s\n' % \
+                ((self.d.getVar('BAD_RECOMMENDATIONS', True) or '').strip(),
+                 (self.d.getVar('NO_RECOMMENDATIONS', True) or '').strip(),
+                 (self.d.getVar('PACKAGE_EXCLUDE', True) or '').strip())
+        open(vars_list_dir, 'w+').write(new_vars_list)
+
+        if old_vars_list != new_vars_list:
+            return True
+
+        return False
+
      def _create(self):
          pkgs_to_install = self.manifest.parse_initial_manifest()
          opkg_pre_process_cmds = self.d.getVar('OPKG_PREPROCESS_COMMANDS', True)
Signature · 1 line
1.8.1.2
author
#9 lore raw

PATCH 5/5 rootfs.py: tweak _multilib_sanity_test for ipk incremental image generation

In reply to Hongxu Jia

The _multilib_sanity_test installs multilib packages in a temporary root fs, and compare with the current image to figure out duplicated file that comes from different packages.

While incremental image generation enabled and the previous image existed, there was a Multilib check error:
...
ERROR: Multilib check error: duplicate files tmp/work/qemux86_64-poky- linux/core-image-minimal/1.0-r0/multilib/lib32/lib/libc.so.6 tmp/work/ qemux86_64-poky-linux/core-image-minimal/1.0-r0/rootfs/lib/libc.so.6 is not the same
...

The reason is the file in current image has been prelinked in previous image generation and the file in a temporary root fs is not prelinked, even though the files come from the same package, the Multilib check considers they are different.

[YOCTO #1894]
---
 meta/lib/oe/rootfs.py | 56 +++++++++++++++++++++++++++++++++++++++++++++++----
 1 file changed, 52 insertions(+), 4 deletions(-)
meta/lib/oe/rootfs.py +52 -4
index 3d7adf9..5054d1e 100644
@@ -3,6 +3,8 @@ from oe.utils import execute_pre_post_process
 from oe.utils import contains as base_contains
 from oe.package_manager import *
 from oe.manifest import *
+import oe.path
+import filecmp
 import shutil
 import os
 import subprocess
@@ -458,13 +460,61 @@ class OpkgRootfs(Rootfs):
 
         bb.utils.remove(self.d.getVar('MULTILIB_TEMP_ROOTFS', True), True)
 
+    def _prelink_file(self, root_dir, filename):
+        bb.note('prelink %s in %s' % (filename, root_dir))
+        prelink_cfg = oe.path.join(root_dir,
+                                   self.d.expand('${sysconfdir}/prelink.conf'))
+        if not os.path.exists(prelink_cfg):
+            shutil.copy(self.d.expand('${STAGING_DIR_NATIVE}${sysconfdir_native}/prelink.conf'),
+                        prelink_cfg)
+
+        cmd_prelink = self.d.expand('${STAGING_DIR_NATIVE}${sbindir_native}/prelink')
+        self._exec_shell_cmd([cmd_prelink,
+                              '--root',
+                              root_dir,
+                              '-amR',
+                              '-N',
+                              '-c',
+                              self.d.expand('${sysconfdir}/prelink.conf')])
+
+    '''
+    Compare two files with the same key twice to see if they came
+    from the same package. If they are not same, they are duplicated
+    and come from different packages.
+    1st: Comapre them directly;
+    2nd: While incremental image creation is enabled, one of the
+         files could be probaly prelinked in the previous image
+         creation and the file has been changed, so we need to
+         prelink the other one and compare them.
+    '''
+    def _file_duplicate(self, key, f1, f2):
+
+        if not os.path.exists(f1) or not os.path.exists(f2):
+            return False
+
+        # f1 is the same with f2, both of them were not prelinked
+        if filecmp.cmp(f1, f2):
+            return False
+
+        if self.image_rootfs not in f1:
+            self._prelink_file(f1.replace(key, ''), f1)
+
+        if self.image_rootfs not in f2:
+            self._prelink_file(f2.replace(key, ''), f2)
+
+        # f1 is the same with f2, both of them were prelinked
+        if filecmp.cmp(f1, f2):
+            return False
+
+        # Duplicated
+        return True
+
     """
     This function was reused from the old implementation.
     See commit: "image.bbclass: Added variables for multilib support." by
     Lianhao Lu.
     """
     def _multilib_sanity_test(self, dirs):
-        import filecmp
 
         allow_replace = self.d.getVar("MULTILIBRE_ALLOW_REP", True)
         if allow_replace is None:
@@ -486,9 +536,7 @@ class OpkgRootfs(Rootfs):
                         if allow_rep.match(key):
                             valid = True
                         else:
-                            if os.path.exists(files[key]) and \
-                               os.path.exists(item) and \
-                               not filecmp.cmp(files[key], item):
+                            if self._file_duplicate(key, files[key], item):
                                 valid = False
                                 bb.fatal("%s duplicate files %s %s is not the same\n" %
                                          (error_prompt, item, files[key]))
Signature · 1 line
1.8.1.2
#10 lore raw

Re: [PATCH 5/5] rootfs.py: tweak _multilib_sanity_test for ipk incremental image generation

Quoted 52 lines · Hongxu Jia

The _multilib_sanity_test installs multilib packages in a temporary root fs, and compare with the current image to figure out duplicated file that comes from different packages.

While incremental image generation enabled and the previous image existed, there was a Multilib check error:
...
ERROR: Multilib check error: duplicate files tmp/work/qemux86_64-poky- linux/core-image-minimal/1.0-r0/multilib/lib32/lib/libc.so.6 tmp/work/ qemux86_64-poky-linux/core-image-minimal/1.0-r0/rootfs/lib/libc.so.6 is not the same
...

The reason is the file in current image has been prelinked in previous image generation and the file in a temporary root fs is not prelinked, even though the files come from the same package, the Multilib check considers they are different.

[YOCTO #1894]
---
 meta/lib/oe/rootfs.py | 56 +++++++++++++++++++++++++++++++++++++++++++++++----
 1 file changed, 52 insertions(+), 4 deletions(-)
meta/lib/oe/rootfs.py +13 -0
index 3d7adf9..5054d1e 100644
@@ -3,6 +3,8 @@ from oe.utils import execute_pre_post_process
 from oe.utils import contains as base_contains
 from oe.package_manager import *
 from oe.manifest import *
+import oe.path
+import filecmp
 import shutil
 import os
 import subprocess
@@ -458,13 +460,61 @@ class OpkgRootfs(Rootfs):
 
         bb.utils.remove(self.d.getVar('MULTILIB_TEMP_ROOTFS', True), True)
 
+    def _prelink_file(self, root_dir, filename):
+        bb.note('prelink %s in %s' % (filename, root_dir))
+        prelink_cfg = oe.path.join(root_dir,
+                                   self.d.expand('${sysconfdir}/prelink.conf'))
+        if not os.path.exists(prelink_cfg):
+            shutil.copy(self.d.expand('${STAGING_DIR_NATIVE}${sysconfdir_native}/prelink.conf'),
+                        prelink_cfg)
+
+        cmd_prelink = self.d.expand('${STAGING_DIR_NATIVE}${sbindir_native}/prelink')
+        self._exec_shell_cmd([cmd_prelink,
+                              '--root',

meta/lib/oe/rootfs.py class OpkgRootfs(Rootfs): · line 483

Quoted 10 lines
meta/lib/oe/rootfs.py +10 -0
+                              root_dir,
+                              '-amR',
+                              '-N',
+                              '-c',
+                              self.d.expand('${sysconfdir}/prelink.conf')])
+
+    '''
+    Compare two files with the same key twice to see if they came
+    from the same package. If they are not same, they are duplicated
+    and come from different packages.

I'm kind of confused by this comment. Doesn't same = duplicate? There might be a small confusion of terms here because the function's behavior is not as the name implies.

meta/lib/oe/rootfs.py class OpkgRootfs(Rootfs): · line 490

Quoted 7 lines
meta/lib/oe/rootfs.py +7 -0
+    1st: Comapre them directly;
+    2nd: While incremental image creation is enabled, one of the
+         files could be probaly prelinked in the previous image
+         creation and the file has been changed, so we need to
+         prelink the other one and compare them.
+    '''
+    def _file_duplicate(self, key, f1, f2):

Shouldn't be better to rename this function to something else in order to avoid confusion? Let's say: _files_are_equal() ?

meta/lib/oe/rootfs.py class OpkgRootfs(Rootfs): · line 497

Quoted 7 lines
meta/lib/oe/rootfs.py +7 -0
+
+        if not os.path.exists(f1) or not os.path.exists(f2):
+            return False
+
+        # f1 is the same with f2, both of them were not prelinked
+        if filecmp.cmp(f1, f2):
+            return False

filecmp.cmp() returns True if files are equal. Hence the confusion:
_file_duplicate() returns False here, if files are equal... I think the logic is a little bit the other way around! :)

meta/lib/oe/rootfs.py class OpkgRootfs(Rootfs): · line 510

Quoted 13 lines
meta/lib/oe/rootfs.py +13 -0
+
+        if self.image_rootfs not in f1:
+            self._prelink_file(f1.replace(key, ''), f1)
+
+        if self.image_rootfs not in f2:
+            self._prelink_file(f2.replace(key, ''), f2)
+
+        # f1 is the same with f2, both of them were prelinked
+        if filecmp.cmp(f1, f2):
+            return False
+
+        # Duplicated
+        return True

here the return value matches the comment and the function name! :)

Quoted 25 lines
meta/lib/oe/rootfs.py +2 -4
+
     """
     This function was reused from the old implementation.
     See commit: "image.bbclass: Added variables for multilib support." by
     Lianhao Lu.
     """
     def _multilib_sanity_test(self, dirs):
-        import filecmp
 
         allow_replace = self.d.getVar("MULTILIBRE_ALLOW_REP", True)
         if allow_replace is None:
@@ -486,9 +536,7 @@ class OpkgRootfs(Rootfs):
                         if allow_rep.match(key):
                             valid = True
                         else:
-                            if os.path.exists(files[key]) and \
-                               os.path.exists(item) and \
-                               not filecmp.cmp(files[key], item):
+                            if self._file_duplicate(key, files[key], item):
                                 valid = False
                                 bb.fatal("%s duplicate files %s %s is not the same\n" %
                                          (error_prompt, item, files[key]))
Signature · 1 line
1.8.1.2
author
#11 lore raw

Re: [PATCH 5/5] rootfs.py: tweak _multilib_sanity_test for ipk incremental image generation

I will rename _file_duplicate with _file_equal, and do the necessary ajustment to avoid confusions.

Thank you for pointing out
V2 incoming

//Hongxu

Quoted 127 lines · Laurentiu Palcu
Quoted 52 lines · Hongxu Jia

The _multilib_sanity_test installs multilib packages in a temporary root fs, and compare with the current image to figure out duplicated file that comes from different packages.

While incremental image generation enabled and the previous image existed, there was a Multilib check error:
...
ERROR: Multilib check error: duplicate files tmp/work/qemux86_64-poky- linux/core-image-minimal/1.0-r0/multilib/lib32/lib/libc.so.6 tmp/work/ qemux86_64-poky-linux/core-image-minimal/1.0-r0/rootfs/lib/libc.so.6 is not the same
...

The reason is the file in current image has been prelinked in previous image generation and the file in a temporary root fs is not prelinked, even though the files come from the same package, the Multilib check considers they are different.

[YOCTO #1894]
---
  meta/lib/oe/rootfs.py | 56 +++++++++++++++++++++++++++++++++++++++++++++++----
  1 file changed, 52 insertions(+), 4 deletions(-)
meta/lib/oe/rootfs.py +13 -0
index 3d7adf9..5054d1e 100644
@@ -3,6 +3,8 @@ from oe.utils import execute_pre_post_process
  from oe.utils import contains as base_contains
  from oe.package_manager import *
  from oe.manifest import *
+import oe.path
+import filecmp
  import shutil
  import os
  import subprocess
@@ -458,13 +460,61 @@ class OpkgRootfs(Rootfs):
 
          bb.utils.remove(self.d.getVar('MULTILIB_TEMP_ROOTFS', True), True)
 
+    def _prelink_file(self, root_dir, filename):
+        bb.note('prelink %s in %s' % (filename, root_dir))
+        prelink_cfg = oe.path.join(root_dir,
+                                   self.d.expand('${sysconfdir}/prelink.conf'))
+        if not os.path.exists(prelink_cfg):
+            shutil.copy(self.d.expand('${STAGING_DIR_NATIVE}${sysconfdir_native}/prelink.conf'),
+                        prelink_cfg)
+
+        cmd_prelink = self.d.expand('${STAGING_DIR_NATIVE}${sbindir_native}/prelink')
+        self._exec_shell_cmd([cmd_prelink,
+                              '--root',

meta/lib/oe/rootfs.py class OpkgRootfs(Rootfs): · line 483

Quoted 10 lines
meta/lib/oe/rootfs.py +10 -0
+                              root_dir,
+                              '-amR',
+                              '-N',
+                              '-c',
+                              self.d.expand('${sysconfdir}/prelink.conf')])
+
+    '''
+    Compare two files with the same key twice to see if they came
+    from the same package. If they are not same, they are duplicated
+    and come from different packages.

I'm kind of confused by this comment. Doesn't same = duplicate? There might be a small confusion of terms here because the function's behavior is not as the name implies.

meta/lib/oe/rootfs.py class OpkgRootfs(Rootfs): · line 490

Quoted 7 lines
meta/lib/oe/rootfs.py +7 -0
+    1st: Comapre them directly;
+    2nd: While incremental image creation is enabled, one of the
+         files could be probaly prelinked in the previous image
+         creation and the file has been changed, so we need to
+         prelink the other one and compare them.
+    '''
+    def _file_duplicate(self, key, f1, f2):

Shouldn't be better to rename this function to something else in order to avoid confusion? Let's say: _files_are_equal() ?

meta/lib/oe/rootfs.py class OpkgRootfs(Rootfs): · line 497

Quoted 7 lines
meta/lib/oe/rootfs.py +7 -0
+
+        if not os.path.exists(f1) or not os.path.exists(f2):
+            return False
+
+        # f1 is the same with f2, both of them were not prelinked
+        if filecmp.cmp(f1, f2):
+            return False

filecmp.cmp() returns True if files are equal. Hence the confusion:
_file_duplicate() returns False here, if files are equal... I think the logic is a little bit the other way around! :)

meta/lib/oe/rootfs.py class OpkgRootfs(Rootfs): · line 510

Quoted 13 lines
meta/lib/oe/rootfs.py +13 -0
+
+        if self.image_rootfs not in f1:
+            self._prelink_file(f1.replace(key, ''), f1)
+
+        if self.image_rootfs not in f2:
+            self._prelink_file(f2.replace(key, ''), f2)
+
+        # f1 is the same with f2, both of them were prelinked
+        if filecmp.cmp(f1, f2):
+            return False
+
+        # Duplicated
+        return True

here the return value matches the comment and the function name! :)

Quoted 25 lines
meta/lib/oe/rootfs.py +2 -4
+
      """
      This function was reused from the old implementation.
      See commit: "image.bbclass: Added variables for multilib support." by
      Lianhao Lu.
      """
      def _multilib_sanity_test(self, dirs):
-        import filecmp
 
          allow_replace = self.d.getVar("MULTILIBRE_ALLOW_REP", True)
          if allow_replace is None:
@@ -486,9 +536,7 @@ class OpkgRootfs(Rootfs):
                          if allow_rep.match(key):
                              valid = True
                          else:
-                            if os.path.exists(files[key]) and \
-                               os.path.exists(item) and \
-                               not filecmp.cmp(files[key], item):
+                            if self._file_duplicate(key, files[key], item):
                                  valid = False
                                  bb.fatal("%s duplicate files %s %s is not the same\n" %
                                           (error_prompt, item, files[key]))
Signature · 1 line
1.8.1.2