Skip to contentSkip to conversation

python-smartpm: Add an attempt install mode

PATCH In review Mark Hatle 2 participants 5 messages Sep 11, 2013

lore
Share & tools
lore

Mark Hatle

author
#0 lore raw

PATCH 1/1 python-smartpm: Add an attempt install mode

[ YOCTO #3723 ]

Add a mode to smart that will allow an installation to continue, instead of failure in the case that one or more items is uninstallable.

Uninstallable packages are simply ignored, and no error is generated.

---
 meta/classes/package_rpm.bbclass                   |  7 +--
 .../python/python-smartpm/smart-attempt.patch      | 59 ++++++++++++++++++++++
 .../python/python-smartpm_1.4.1.bb                 |  1 +
 3 files changed, 62 insertions(+), 5 deletions(-)
 create mode 100644 meta/recipes-devtools/python/python-smartpm/smart-attempt.patch
meta/classes/package_rpm.bbclass +2 -5
index 8ab42c4..699d0f2 100644
@@ -472,11 +472,8 @@ EOF
 		echo "Attempting $pkgs_to_install"
 		echo "Note: see `dirname ${BB_LOGFILE}`/log.do_${task}_attemptonly.${PID}"
 		translate_oe_to_smart ${sdk_mode} --attemptonly $package_attemptonly
-		for each_pkg in $pkgs_to_install ;  do
-			# We need to try each package individually as a single dependency failure
-			# will break the whole set otherwise.
-			smart --data-dir=${target_rootfs}/var/lib/smart install -y $each_pkg >> "`dirname ${BB_LOGFILE}`/log.do_${task}_attemptonly.${PID}" 2>&1 || true
-		done
+		echo "Attempting $pkgs_to_install" >> "`dirname ${BB_LOGFILE}`/log.do_${task}_attemptonly.${PID}"
+		smart --data-dir=${target_rootfs}/var/lib/smart install --attempt -y ${pkgs_to_install} >> "`dirname ${BB_LOGFILE}`/log.do_${task}_attemptonly.${PID}" 2>&1
 	fi
 }
 
meta/recipes-devtools/python/python-smartpm/smart-attempt.patch new file +59 -0

59 lines not on this page · raw message

meta/recipes-devtools/python/python-smartpm_1.4.1.bb +1 -0
index 5d673b9..f406d69 100644
@@ -30,6 +30,7 @@ SRC_URI = "\
           file://smart-flag-ignore-recommends.patch \
           file://smart-flag-exclude-packages.patch \
           file://smart-config-ignore-all-recommends.patch \
+          file://smart-attempt.patch \
           "
 
 SRC_URI[md5sum] = "573ef32ba177a6b3c4bf7ef04873fcb6"
Signature · 1 line
1.8.1.2.545.g2f19ada
author
#1 lore raw

Time measurements with:

MACHINE = "qemux86"
PACKAGE_CLASSES = "package_rpm"
EXTRA_IMAGE_FEATURES = "dev-pkgs staticdev-pkgs doc-pkgs dbg-pkgs ptest-pkgs" BB_NUMBER_THREADS ?= "8"
PARALLEL_MAKE ?= "-j 8"

image: core-image-sato

Quad Core i7 workstation...

With all of the recipes built, and just executing the do_rootfs:

Before this change:

 real    26m2.541s
 user    21m8.458s
 sys     7m9.683s

After this change:

 real    14m43.436s
 user    15m57.525s
 sys     5m16.412s

This is a significant performance boost when using the 'complementary' installs and attemptonly installs. The performance is by doing the complementary installs as a batch instead of one at a time, ignoring failures.

--Mark

Quoted 111 lines · Mark Hatle

[ YOCTO #3723 ]

Add a mode to smart that will allow an installation to continue, instead of failure in the case that one or more items is uninstallable.

Uninstallable packages are simply ignored, and no error is generated.

---
  meta/classes/package_rpm.bbclass                   |  7 +--
  .../python/python-smartpm/smart-attempt.patch      | 59 ++++++++++++++++++++++
  .../python/python-smartpm_1.4.1.bb                 |  1 +
  3 files changed, 62 insertions(+), 5 deletions(-)

create mode 100644 meta/recipes-devtools/python/python-smartpm/smart-attempt.patch

meta/classes/package_rpm.bbclass +2 -5
index 8ab42c4..699d0f2 100644
@@ -472,11 +472,8 @@ EOF
  		echo "Attempting $pkgs_to_install"
  		echo "Note: see `dirname ${BB_LOGFILE}`/log.do_${task}_attemptonly.${PID}"
  		translate_oe_to_smart ${sdk_mode} --attemptonly $package_attemptonly
-		for each_pkg in $pkgs_to_install ;  do
-			# We need to try each package individually as a single dependency failure
-			# will break the whole set otherwise.
-			smart --data-dir=${target_rootfs}/var/lib/smart install -y $each_pkg >> "`dirname ${BB_LOGFILE}`/log.do_${task}_attemptonly.${PID}" 2>&1 || true
-		done
+		echo "Attempting $pkgs_to_install" >> "`dirname ${BB_LOGFILE}`/log.do_${task}_attemptonly.${PID}"
+		smart --data-dir=${target_rootfs}/var/lib/smart install --attempt -y ${pkgs_to_install} >> "`dirname ${BB_LOGFILE}`/log.do_${task}_attemptonly.${PID}" 2>&1
  	fi
  }
 
meta/recipes-devtools/python/python-smartpm/smart-attempt.patch new file +59 -0
new file mode 100644
index 0000000..0d603d3
@@ -0,0 +1,59 @@
+Add a mechanism to attempt the install operation, w/o failing.
+
+For complementary and 'attemptonly' packages, we need a way to instruct smart to
+try to install, but ignore any failures.
+
+This option only works for the install operation.
+
+Upstream-Status: Pending
+
+Signed-off-by: Mark Hatle <[email protected]>
+
+Index: smart-1.4.1/smart/commands/install.py
+===================================================================
+--- smart-1.4.1.orig/smart/commands/install.py
++++ smart-1.4.1/smart/commands/install.py
+@@ -50,6 +50,8 @@ def option_parser():
+     parser = OptionParser(usage=USAGE,
+                           description=DESCRIPTION,
+                           examples=EXAMPLES)
++    parser.add_option("--attempt", action="store_true",
++                      help=_("attempt to install packages, ignore failures"))
+     parser.add_option("--stepped", action="store_true",
+                       help=_("split operation in steps"))
+     parser.add_option("--urls", action="store_true",
+@@ -80,6 +82,9 @@ def main(ctrl, opts):
+     if not opts.args:
+         raise Error, _("no package(s) given")
+
++    if opts.attempt:
++	sysconf.set("attempt-install", True, soft=True)
++
+     if opts.explain:
+         sysconf.set("explain-changesets", True, soft=True)
+
+Index: smart-1.4.1/smart/transaction.py
+===================================================================
+--- smart-1.4.1.orig/smart/transaction.py
++++ smart-1.4.1/smart/transaction.py
+@@ -1216,9 +1216,17 @@ class Transaction(object):
+                     else:
+                         op = REMOVE
+                 if op is INSTALL or op is REINSTALL:
+-                    self._install(pkg, changeset, locked, pending)
+-                    if pkg in changeset:
+-                        changeset.setRequested(pkg, True)
++                    try:
++                        self._install(pkg, changeset, locked, pending)
++                        if pkg in changeset:
++                            changeset.setRequested(pkg, True)
++                    except Failed, e:
++                        if sysconf.has("attempt-install", soft=True):
++                            if pkg in changeset:
++                                del changeset[pkg]
++                            continue
++                        else:
++                            raise Failed, e
+                 elif op is REMOVE:
+                     self._remove(pkg, changeset, locked, pending)
+                 elif op is UPGRADE:
meta/recipes-devtools/python/python-smartpm_1.4.1.bb +1 -0
index 5d673b9..f406d69 100644
@@ -30,6 +30,7 @@ SRC_URI = "\
            file://smart-flag-ignore-recommends.patch \
            file://smart-flag-exclude-packages.patch \
            file://smart-config-ignore-all-recommends.patch \
+          file://smart-attempt.patch \
            "
 
  SRC_URI[md5sum] = "573ef32ba177a6b3c4bf7ef04873fcb6"
#2 lore raw

Hi Mark,

Quoted 20 lines · On Tuesday 10 September 2013 21:04:29 Mark Hatle wrote:

Time measurements with:

MACHINE = "qemux86"
PACKAGE_CLASSES = "package_rpm"
EXTRA_IMAGE_FEATURES = "dev-pkgs staticdev-pkgs doc-pkgs dbg-pkgs ptest-pkgs" BB_NUMBER_THREADS ?= "8"
PARALLEL_MAKE ?= "-j 8"

image: core-image-sato

Quad Core i7 workstation...

With all of the recipes built, and just executing the do_rootfs:

Before this change:

  real    26m2.541s
  user    21m8.458s
  sys     7m9.683s

After this change:

Quoted 8 lines
  real    14m43.436s
  user    15m57.525s
  sys     5m16.412s

This is a significant performance boost when using the 'complementary' installs and attemptonly installs. The performance is by doing the complementary installs as a batch instead of one at a time, ignoring failures.

This is a significant improvement, well done! My concern about not catching failures stands though, which is why I hadn't implemented this yet. How do we ensure that genuine failures are caught with this mode?

Cheers,
Paul

Signature · 2 lines
Paul Eggleton
Intel Open Source Technology Centre
author
#3 lore raw
Quoted 27 lines · Paul Eggleton

Hi Mark,

Quoted 24 lines · On Tuesday 10 September 2013 21:04:29 Mark Hatle wrote:

Time measurements with:

MACHINE = "qemux86"
PACKAGE_CLASSES = "package_rpm"
EXTRA_IMAGE_FEATURES = "dev-pkgs staticdev-pkgs doc-pkgs dbg-pkgs ptest-pkgs" BB_NUMBER_THREADS ?= "8"
PARALLEL_MAKE ?= "-j 8"

image: core-image-sato

Quad Core i7 workstation...

With all of the recipes built, and just executing the do_rootfs:

Before this change:

   real    26m2.541s
   user    21m8.458s
   sys     7m9.683s

After this change:

   real    14m43.436s
   user    15m57.525s
   sys     5m16.412s
Quoted 8 lines
Quoted 4 lines

This is a significant performance boost when using the 'complementary' installs and attemptonly installs. The performance is by doing the complementary installs as a batch instead of one at a time, ignoring failures.

This is a significant improvement, well done! My concern about not catching failures stands though, which is why I hadn't implemented this yet. How do we ensure that genuine failures are caught with this mode?

The code only ignores failures in the resolution of the dependencies. As far as I'm aware, this is the correct behavior in the attempt only mode(s). If the dependencies can't be resolved for some reason, we simply throw away that item and move on.

Any errors that occur during package installation -will- now cause a failure to exist, and they will be logged to the attemptonly log-file.

In the previous implementation, any error when installing an attemptonly package would be ignored, even if it turned out to be an installation problem and not a dependency issue. So I think this is an improvement as well.

(For some of my testing I intentionally created dependency errors by adding things to the PACKAGE_EXCLUDE, and the system did what it was supposed to. Ignored the items that couldn't be installed.)

--Mark

Quoted 3 lines

Cheers,
Paul

#4 lore raw
Quoted 44 lines · On Wednesday 11 September 2013 08:47:33 Mark Hatle wrote:
Quoted 30 lines · Paul Eggleton
Quoted 20 lines · On Tuesday 10 September 2013 21:04:29 Mark Hatle wrote:

Time measurements with:

MACHINE = "qemux86"
PACKAGE_CLASSES = "package_rpm"
EXTRA_IMAGE_FEATURES = "dev-pkgs staticdev-pkgs doc-pkgs dbg-pkgs ptest-pkgs" BB_NUMBER_THREADS ?= "8"
PARALLEL_MAKE ?= "-j 8"

image: core-image-sato

Quad Core i7 workstation...

With all of the recipes built, and just executing the do_rootfs:

Before this change:

   real    26m2.541s
   user    21m8.458s
   sys     7m9.683s

After this change:

Quoted 8 lines
   real    14m43.436s
   user    15m57.525s
   sys     5m16.412s

This is a significant performance boost when using the 'complementary' installs and attemptonly installs. The performance is by doing the complementary installs as a batch instead of one at a time, ignoring failures.

Quoted 4 lines

This is a significant improvement, well done! My concern about not catching
failures stands though, which is why I hadn't implemented this yet. How do we ensure that genuine failures are caught with this mode?

The code only ignores failures in the resolution of the dependencies. As far as I'm aware, this is the correct behavior in the attempt only mode(s).
If the dependencies can't be resolved for some reason, we simply throw
away that item and move on.

Any errors that occur during package installation -will- now cause a failure to exist, and they will be logged to the attemptonly log-file.

Quoted 8 lines

In the previous implementation, any error when installing an attemptonly package would be ignored, even if it turned out to be an installation problem and not a dependency issue. So I think this is an improvement as well.

(For some of my testing I intentionally created dependency errors by adding things to the PACKAGE_EXCLUDE, and the system did what it was supposed to. Ignored the items that couldn't be installed.)

Sounds like a great improvement. Thanks!

Cheers,
Paul

Signature · 2 lines
Paul Eggleton
Intel Open Source Technology Centre