fix smart install with --attempt failed

Test steps:

1. vim local.conf
...
IMAGE_INSTALL_append = "shadow man-pages hdparm"
EXTRA_IMAGE_FEATURES += "doc-pkgs"
...

2. bitbake core-image-minimal
...
|WARNING: Unable attempt to install packages. Command '...' returned 1:
|error: file /usr/share/man/man5/passwd.5 from install of shadow-doc-4.2.1-r0.i586 conflicts with file from package man-pages-3.71-r0.i586
|error: file /usr/share/man/man3/getspnam.3 from install of shadow-doc-4.2.1-r0.i586 conflicts with file from package man-pages-3.71-r0.i586

|WARNING: Attempt to install again:...
|NOTE: Tasks Summary: Attempted 2198 tasks of which 2172 didn't need to be rerun and all succeeded.
...

The rest doc packages installed successfully.

//Hongxu

The following changes since commit 8ac8eca2e3bd8c78e2b31ea974930ed0243258a3:

build-appliance-image: Update to dizzy head revision (2014-09-23 22:10:26 +0100)

are available in the git repository at:

git://git.pokylinux.org/poky-contrib hongxu/fix-attempt-install
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=hongxu/fix-attempt-install

Hongxu Jia (2):
rootfs.py: catch inner warn message package_manager.py: ignore failed smart install with --attempt

 meta/lib/oe/package_manager.py | 38 +++++++++++++++++++++++++++++++++++++-
 meta/lib/oe/rootfs.py          | 20 +++++++++++++++++++-
 2 files changed, 56 insertions(+), 2 deletions(-)

1.9.1

Package managements (smart/apt-get/opkg-cl) generate some warn messages to stdout, and we need to catch them and output by bb.warn.

Here is an example, while invoking smart to attempt install doc packages, if install failed, it generates warn message to stdout.
...
|warning: Can't install util-linux-doc-2.24.2-r1@i586: Can't install util-linux-doc-2.24.2-r1@i586: no package provides info
...

The fix catches it and outputs:
...

|WARNING: log_check: There is a warn message in the logfile
|WARNING: log_check: Matched keyword: [warn]
|WARNING: log_check: warning: Can't install util-linux-doc-2.24.2-r1@

i586: Can't install util-linux-doc-2.24.2-r1@i586: no package provides info
...

---
 meta/lib/oe/rootfs.py | 20 +++++++++++++++++++-
 1 file changed, 19 insertions(+), 1 deletion(-)

index 3d8ae81..c4735f2 100644
@@ -347,7 +347,21 @@ class RpmRootfs(Rootfs):
         # already saved in /etc/rpm-postinsts
         pass
 
-    def _log_check(self):
+    def _log_check_warn(self):
+        r = re.compile('(warn|Warn)')
+        log_path = self.d.expand("${T}/log.do_rootfs")
+        with open(log_path, 'r') as log:
+            for line in log.read().split('\n'):
+                if 'log_check' in line:
+                    continue
+
+                m = r.search(line)
+                if m:
+                    bb.warn('log_check: There is a warn message in the logfile')
+                    bb.warn('log_check: Matched keyword: [%s]' % m.group())
+                    bb.warn('log_check: %s\n' % line)
+
+    def _log_check_error(self):
         r = re.compile('(unpacking of archive failed|Cannot find package|exit 1|ERR|Fail)')
         log_path = self.d.expand("${T}/log.do_rootfs")
         with open(log_path, 'r') as log:

@@ -370,6 +384,10 @@ class RpmRootfs(Rootfs):
                 if found_error == 6:
                     bb.fatal(message)
 
+    def _log_check(self):
+        self._log_check_warn()
+        self._log_check_error()
+
     def _handle_intercept_failure(self, registered_pkgs):
         rpm_postinsts_dir = self.image_rootfs + self.d.expand('${sysconfdir}/rpm-postinsts/')
         bb.utils.mkdirhier(rpm_postinsts_dir)

1.9.1

Paul's patch added checking in smart to ignore failed install with --attempt
...
commit e9c1191da4f84c9743b053e7030ab28b48f57c44

Author: Paul Eggleton <[email protected]>
Date:   Thu Feb 6 13:39:01 2014 +0000

python-smartpm: really ignore conflicts during install with --attempt
...

But if the failure is generated by rpm which smart doesn't resolve, we check the output of smart, filter the failed packages, and attempt to install the rest.

Here is an example, assume there are two packages which have identical file name but different content, that cause confliction. If we install one without --attempt, after it installed and install another with --attempt, the rpm will generate an error which smart does not resolve:
...
|error: file /usr/share/man/man5/passwd.5 from install of shadow-doc-4.2.1-r0.i586 conflicts with file from package man-pages-3.71-r0.i586
|error: file /usr/share/man/man3/getspnam.3 from install of shadow-doc-4.2.1-r0.i586 conflicts with file from package man-pages-3.71-r0.i586
...

In this situation, remove failed packages, and attempt to install the rest.

[YOCTO #6769]

---
 meta/lib/oe/package_manager.py | 38 +++++++++++++++++++++++++++++++++++++-
 1 file changed, 37 insertions(+), 1 deletion(-)

index b3b3b2d..20997d3 100644
@@ -955,7 +955,43 @@ class RpmPM(PackageManager):
             output = subprocess.check_output(cmd.split(), stderr=subprocess.STDOUT)
             bb.note(output)
         except subprocess.CalledProcessError as e:
-            bb.fatal("Unable to install packages. Command '%s' "
+            if not attempt_only:
+                bb.fatal("Unable to install packages. Command '%s' "
+                         "returned %d:\n%s" % (cmd, e.returncode, e.output))
+            else:
+                bb.warn("Unable attempt to install packages. Command '%s' "
+                         "returned %d:\n%s" % (cmd, e.returncode, e.output))
+
+                # Remove failed packages, and attempt to install the rest
+                reinstall_pkgs = pkgs[:]
+                for line in e.output.split("\n"):
+                    if line.strip() == "":
+                        continue
+                    for pkg in pkgs[:]:
+                        name = pkg.split("@")[0]
+                        arch = pkg.split("@")[1]
+                        r = re.compile(r'%s\S+%s' % (name, arch))
+                        if r.search(line):
+                            pkgs.remove(pkg)
+
+                bb.warn("Attempt to install again: %s" % ' '.join(pkgs))
+                self._attempt_install(pkgs)
+
+    '''
+    Attempt to install spkgs, the spkgs is a pkg list with smart format
+    '''
+    def _attempt_install(self, spkgs):
+        if len(spkgs) == 0:
+            return
+
+        bb.note("Attempt to install the following packages: %s" % ' '.join(spkgs))
+        cmd = "%s %s install --attempt -y %s" % \
+              (self.smart_cmd, self.smart_opt, ' '.join(spkgs))
+        try:
+            output = subprocess.check_output(cmd.split(), stderr=subprocess.STDOUT)
+            bb.note(output)
+        except subprocess.CalledProcessError as e:
+            bb.warn("Unable attempt to install packages. Command '%s' "
                      "returned %d:\n%s" % (cmd, e.returncode, e.output))
 
     '''

1.9.1

Hi Hongxu,

Apologies for the delay in reviewing this.

I see what you're trying to do here. However I am concerned about the other kinds of failure cases we might end up obscuring with this kind of approach. My understanding of attemptonly is that it should ignore conflicts, but any other kind of error should trigger build failure - otherwise you can miss genuine errors that need to be fixed.

If possible I would prefer if we would detect and handle this kind of conflict within smart itself and allow it to continue rather than detecting the failure and retrying externally.

Mark (on CC), any opinions?

Cheers,
Paul

Paul Eggleton
Intel Open Source Technology Centre