python-smartpm: Add an attempt install mode
PATCH 1/1 python-smartpm: Add an attempt install mode
[ YOCTO #3723 ]
Add a mode to smart that will allow an installation to continue, instead of failure in the case that one or more items is uninstallable.
Uninstallable packages are simply ignored, and no error is generated.
--- meta/classes/package_rpm.bbclass | 7 +-- .../python/python-smartpm/smart-attempt.patch | 59 ++++++++++++++++++++++ .../python/python-smartpm_1.4.1.bb | 1 + 3 files changed, 62 insertions(+), 5 deletions(-) create mode 100644 meta/recipes-devtools/python/python-smartpm/smart-attempt.patch
meta/classes/package_rpm.bbclass +2 -5
@@ -472,11 +472,8 @@ EOF echo "Attempting $pkgs_to_install" echo "Note: see `dirname ${BB_LOGFILE}`/log.do_${task}_attemptonly.${PID}" translate_oe_to_smart ${sdk_mode} --attemptonly $package_attemptonly - for each_pkg in $pkgs_to_install ; do - # We need to try each package individually as a single dependency failure - # will break the whole set otherwise. - smart --data-dir=${target_rootfs}/var/lib/smart install -y $each_pkg >> "`dirname ${BB_LOGFILE}`/log.do_${task}_attemptonly.${PID}" 2>&1 || true - done + echo "Attempting $pkgs_to_install" >> "`dirname ${BB_LOGFILE}`/log.do_${task}_attemptonly.${PID}" + smart --data-dir=${target_rootfs}/var/lib/smart install --attempt -y ${pkgs_to_install} >> "`dirname ${BB_LOGFILE}`/log.do_${task}_attemptonly.${PID}" 2>&1 fi }
meta/recipes-devtools/python/python-smartpm/smart-attempt.patch new file +59 -0
59 lines not on this page · raw message
meta/recipes-devtools/python/python-smartpm_1.4.1.bb +1 -0
@@ -30,6 +30,7 @@ SRC_URI = "\ file://smart-flag-ignore-recommends.patch \ file://smart-flag-exclude-packages.patch \ file://smart-config-ignore-all-recommends.patch \ + file://smart-attempt.patch \ " SRC_URI[md5sum] = "573ef32ba177a6b3c4bf7ef04873fcb6"
Signature · 1 line
1.8.1.2.545.g2f19ada
Time measurements with:
MACHINE = "qemux86"PACKAGE_CLASSES = "package_rpm"EXTRA_IMAGE_FEATURES = "dev-pkgs staticdev-pkgs doc-pkgs dbg-pkgs ptest-pkgs" BB_NUMBER_THREADS ?= "8"PARALLEL_MAKE ?= "-j 8"
image: core-image-sato
Quad Core i7 workstation...
With all of the recipes built, and just executing the do_rootfs:
Before this change:
real 26m2.541s user 21m8.458s sys 7m9.683s
After this change:
real 14m43.436s user 15m57.525s sys 5m16.412s
This is a significant performance boost when using the 'complementary' installs and attemptonly installs. The performance is by doing the complementary installs as a batch instead of one at a time, ignoring failures.
--Mark
Quoted 111 lines · Mark Hatle
[ YOCTO #3723 ]
Add a mode to smart that will allow an installation to continue, instead of failure in the case that one or more items is uninstallable.
Uninstallable packages are simply ignored, and no error is generated.
Signed-off-by Mark Hatle <[email protected]>--- meta/classes/package_rpm.bbclass | 7 +-- .../python/python-smartpm/smart-attempt.patch | 59 ++++++++++++++++++++++ .../python/python-smartpm_1.4.1.bb | 1 + 3 files changed, 62 insertions(+), 5 deletions(-)create mode 100644
meta/recipes-devtools/python/python-smartpm/smart-attempt.patchmeta/classes/package_rpm.bbclass +2
-5@@ -472,11 +472,8 @@ EOF echo "Attempting $pkgs_to_install" echo "Note: see `dirname ${BB_LOGFILE}`/log.do_${task}_attemptonly.${PID}" translate_oe_to_smart ${sdk_mode} --attemptonly $package_attemptonly - for each_pkg in $pkgs_to_install ; do - # We need to try each package individually as a single dependency failure - # will break the whole set otherwise. - smart --data-dir=${target_rootfs}/var/lib/smart install -y $each_pkg >> "`dirname ${BB_LOGFILE}`/log.do_${task}_attemptonly.${PID}" 2>&1 || true - done + echo "Attempting $pkgs_to_install" >> "`dirname ${BB_LOGFILE}`/log.do_${task}_attemptonly.${PID}" + smart --data-dir=${target_rootfs}/var/lib/smart install --attempt -y ${pkgs_to_install} >> "`dirname ${BB_LOGFILE}`/log.do_${task}_attemptonly.${PID}" 2>&1 fi }meta/recipes-devtools/python/python-smartpm/smart-attempt.patch new file +59
-0@@ -0,0 +1,59 @@ +Add a mechanism to attempt the install operation, w/o failing. + +For complementary and 'attemptonly' packages, we need a way to instruct smart to +try to install, but ignore any failures. + +This option only works for the install operation. + +Upstream-Status: Pending + +Signed-off-by: Mark Hatle <[email protected]> + +Index: smart-1.4.1/smart/commands/install.py +=================================================================== +--- smart-1.4.1.orig/smart/commands/install.py ++++ smart-1.4.1/smart/commands/install.py +@@ -50,6 +50,8 @@ def option_parser(): + parser = OptionParser(usage=USAGE, + description=DESCRIPTION, + examples=EXAMPLES) ++ parser.add_option("--attempt", action="store_true", ++ help=_("attempt to install packages, ignore failures")) + parser.add_option("--stepped", action="store_true", + help=_("split operation in steps")) + parser.add_option("--urls", action="store_true", +@@ -80,6 +82,9 @@ def main(ctrl, opts): + if not opts.args: + raise Error, _("no package(s) given") + ++ if opts.attempt: ++ sysconf.set("attempt-install", True, soft=True) ++ + if opts.explain: + sysconf.set("explain-changesets", True, soft=True) + +Index: smart-1.4.1/smart/transaction.py +=================================================================== +--- smart-1.4.1.orig/smart/transaction.py ++++ smart-1.4.1/smart/transaction.py +@@ -1216,9 +1216,17 @@ class Transaction(object): + else: + op = REMOVE + if op is INSTALL or op is REINSTALL: +- self._install(pkg, changeset, locked, pending) +- if pkg in changeset: +- changeset.setRequested(pkg, True) ++ try: ++ self._install(pkg, changeset, locked, pending) ++ if pkg in changeset: ++ changeset.setRequested(pkg, True) ++ except Failed, e: ++ if sysconf.has("attempt-install", soft=True): ++ if pkg in changeset: ++ del changeset[pkg] ++ continue ++ else: ++ raise Failed, e + elif op is REMOVE: + self._remove(pkg, changeset, locked, pending) + elif op is UPGRADE:meta/recipes-devtools/python/python-smartpm_1.4.1.bb +1
-0@@ -30,6 +30,7 @@ SRC_URI = "\ file://smart-flag-ignore-recommends.patch \ file://smart-flag-exclude-packages.patch \ file://smart-config-ignore-all-recommends.patch \ + file://smart-attempt.patch \ " SRC_URI[md5sum] = "573ef32ba177a6b3c4bf7ef04873fcb6"
Hi Mark,
Quoted 20 lines · On Tuesday 10 September 2013 21:04:29 Mark Hatle wrote:
Time measurements with:
MACHINE = "qemux86"
PACKAGE_CLASSES= "package_rpm"EXTRA_IMAGE_FEATURES= "dev-pkgs staticdev-pkgs doc-pkgs dbg-pkgs ptest-pkgs"BB_NUMBER_THREADS?= "8"PARALLEL_MAKE?= "-j 8"image: core-image-sato
Quad Core i7 workstation...
With all of the recipes built, and just executing the
do_rootfs:Before this change:
real 26m2.541s user 21m8.458s sys 7m9.683sAfter this change:
Quoted 8 lines
real 14m43.436s user 15m57.525s sys 5m16.412sThis is a significant performance boost when using the 'complementary' installs and attemptonly installs. The performance is by doing the complementary installs as a batch instead of one at a time, ignoring failures.
This is a significant improvement, well done! My concern about not catching failures stands though, which is why I hadn't implemented this yet. How do we ensure that genuine failures are caught with this mode?
Cheers,
Paul
Signature · 2 lines
Paul Eggleton Intel Open Source Technology Centre
Quoted 27 lines · Paul Eggleton
Hi Mark,
Quoted 24 lines · On Tuesday 10 September 2013 21:04:29 Mark Hatle wrote:
Time measurements with:
MACHINE = "qemux86"
PACKAGE_CLASSES= "package_rpm"EXTRA_IMAGE_FEATURES= "dev-pkgs staticdev-pkgs doc-pkgs dbg-pkgs ptest-pkgs"BB_NUMBER_THREADS?= "8"PARALLEL_MAKE?= "-j 8"image: core-image-sato
Quad Core i7 workstation...
With all of the recipes built, and just executing the
do_rootfs:Before this change:
real 26m2.541s user 21m8.458s sys 7m9.683sAfter this change:
real 14m43.436s user 15m57.525s sys 5m16.412s
Quoted 8 lines
Quoted 4 lines
This is a significant performance boost when using the 'complementary' installs and attemptonly installs. The performance is by doing the complementary installs as a batch instead of one at a time, ignoring failures.
This is a significant improvement, well done! My concern about not catching failures stands though, which is why I hadn't implemented this yet. How do we ensure that genuine failures are caught with this mode?
The code only ignores failures in the resolution of the dependencies. As far as I'm aware, this is the correct behavior in the attempt only mode(s). If the dependencies can't be resolved for some reason, we simply throw away that item and move on.
Any errors that occur during package installation -will- now cause a failure to exist, and they will be logged to the attemptonly log-file.
In the previous implementation, any error when installing an attemptonly package would be ignored, even if it turned out to be an installation problem and not a dependency issue. So I think this is an improvement as well.
(For some of my testing I intentionally created dependency errors by adding things to the PACKAGE_EXCLUDE, and the system did what it was supposed to. Ignored the items that couldn't be installed.)
--Mark
Quoted 3 lines
Cheers,
Paul
Quoted 44 lines · On Wednesday 11 September 2013 08:47:33 Mark Hatle wrote:
Quoted 30 lines · Paul Eggleton
Quoted 20 lines · On Tuesday 10 September 2013 21:04:29 Mark Hatle wrote:
Time measurements with:
MACHINE = "qemux86"
PACKAGE_CLASSES= "package_rpm"EXTRA_IMAGE_FEATURES= "dev-pkgs staticdev-pkgs doc-pkgs dbg-pkgs ptest-pkgs"BB_NUMBER_THREADS?= "8"PARALLEL_MAKE?= "-j 8"image: core-image-sato
Quad Core i7 workstation...
With all of the recipes built, and just executing the
do_rootfs:Before this change:
real 26m2.541s user 21m8.458s sys 7m9.683sAfter this change:
Quoted 8 lines
real 14m43.436s user 15m57.525s sys 5m16.412sThis is a significant performance boost when using the 'complementary' installs and attemptonly installs. The performance is by doing the complementary installs as a batch instead of one at a time, ignoring failures.
Quoted 4 lines
This is a significant improvement, well done! My concern about not catching
failures stands though, which is why I hadn't implemented this yet. How do we ensure that genuine failures are caught with this mode?The code only ignores failures in the resolution of the dependencies. As far as I'm aware, this is the correct behavior in the attempt only mode(s).
If the dependencies can't be resolved for some reason, we simply throw
away that item and move on.Any errors that occur during package installation -will- now cause a failure to exist, and they will be logged to the attemptonly log-file.
Quoted 8 lines
In the previous implementation, any error when installing an attemptonly package would be ignored, even if it turned out to be an installation problem and not a dependency issue. So I think this is an improvement as well.
(For some of my testing I intentionally created dependency errors by adding things to the
PACKAGE_EXCLUDE, and the system did what it was supposed to. Ignored the items that couldn't be installed.)
Sounds like a great improvement. Thanks!
Cheers,
Paul
Signature · 2 lines
Paul Eggleton Intel Open Source Technology Centre