wic dependency and rootfs updates

PATCH New Tom Zanussi 1 participant 11 messages Oct 22, 2013

Share & tools

Tom Zanussi

author

2013-10-22 14:24 UTC #0 lore raw

cover wic dependency and rootfs updates

This patchset mainly removes unnecessary run-time python module and executable dependencies for mic functionality we currently don't use. I didn't originally hit these myself because I did development on Fedora and Ubuntu users hit these - I've tested the resulting patchset on both Fedora and Ubuntu with the affected packages removed and don't see any problems.

I've kept them as a series of changes rather than glomming them all together because at some point we will want to restore these dependencies on a case-by-case basis and it's better to have the changes as individually explained commits.

It also removes and unncessary pseudo export as discussed in a previous thread, and moves the setup to the top-level rootfs generation function.

The following changes since commit 505114c13d29799a25379f7ce720d632616d8776:

wic: add pseudo to the populate-extfs step (2013-10-18 16:03:39 +0100)

are available in the git repository at:

git://git.yoctoproject.org/poky-contrib.git tzanussi/wic-fixes-3
http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=tzanussi/wic-fixes-3

Tom Zanussi (10):
wic: Remove dependency on rt_util module
wic: eliminate module checks
wic: Remove dependency on rpmmisc
wic: Remove dependency on myurlgrab module
wic: remove rpm warning code from BackendPlugin
wic: Remove rpm and grabber dependencies from BaseImageCreator
wic: Remove rpmmisc call from livecd
wic: Remove binary dependencies
wic: Make find_binary_path() more user-friendly
wic: Update and generalize pseudo setup for rootfs generation

 scripts/lib/mic/creator.py                         |  4 ++--
 scripts/lib/mic/imager/baseimager.py               |  4 +---
 scripts/lib/mic/imager/livecd.py                   |  4 ++--
 .../lib/mic/kickstart/custom_commands/partition.py | 28 ++++++++++++----------
 scripts/lib/mic/pluginbase.py                      |  5 ----
 scripts/lib/mic/plugins/imager/direct_plugin.py    |  2 +-
 scripts/lib/mic/plugins/imager/fs_plugin.py        |  2 +-
 scripts/lib/mic/plugins/imager/livecd_plugin.py    |  2 +-
 scripts/lib/mic/plugins/imager/liveusb_plugin.py   |  2 +-
 scripts/lib/mic/plugins/imager/loop_plugin.py      |  2 +-
 scripts/lib/mic/plugins/imager/raw_plugin.py       |  2 +-
 scripts/lib/mic/utils/fs_related.py                |  5 +++-
 scripts/lib/mic/utils/misc.py                      |  2 --
 scripts/lib/mic/utils/partitionedfs.py             |  4 ----
 scripts/wic                                        | 10 --------
 15 files changed, 30 insertions(+), 48 deletions(-)

Signature · 1 line

1.8.3.1

Tom Zanussi

author

2013-10-22 14:24 UTC #2 lore raw

PATCH 2/10 wic: eliminate module checks

In reply to Tom Zanussi

We're removing all external dependencies including rpm and urlgrabber, so we don't need this check.

Signed-off-by Tom Zanussi <[email protected]>

---
 scripts/wic | 10 ----------
 1 file changed, 10 deletions(-)

scripts/wic +0 ~~-10~~

index 08473d3..b6fd16c 100755
@@ -38,16 +38,6 @@ import optparse
 import logging
 
 # External modules
-try:
-    import rpm
-    import urlgrabber
-except ImportError:
-    print("ERROR: failed to import necessary modules.")
-    print("Please ensure the following modules are installed:")
-    print("\trpm");
-    print("\turlgrabber");
-    sys.exit(1)
-
 scripts_path = os.path.abspath(os.path.dirname(os.path.abspath(sys.argv[0])))
 lib_path = scripts_path + '/lib'
 sys.path = sys.path + [lib_path]

Signature · 1 line

1.8.3.1

Tom Zanussi

author

2013-10-22 14:24 UTC #1 lore raw

PATCH 1/10 wic: Remove dependency on rt_util module

In reply to Tom Zanussi

rt_util contains bootstrap_mic(), which imports rpm and other things we don't need because we don't do bootstrap i.e. runtime (set in wic.conf) is always set to 'native', which means use what's on the local host.

bootstrap mode is for downloading and installing rpms that wic needs, which we may want to implement later; for now, we just want to use what's local.

Signed-off-by Tom Zanussi <[email protected]>

---
 scripts/lib/mic/creator.py                       | 2 +-
 scripts/lib/mic/plugins/imager/direct_plugin.py  | 2 +-
 scripts/lib/mic/plugins/imager/fs_plugin.py      | 2 +-
 scripts/lib/mic/plugins/imager/livecd_plugin.py  | 2 +-
 scripts/lib/mic/plugins/imager/liveusb_plugin.py | 2 +-
 scripts/lib/mic/plugins/imager/loop_plugin.py    | 2 +-
 scripts/lib/mic/plugins/imager/raw_plugin.py     | 2 +-
 7 files changed, 7 insertions(+), 7 deletions(-)

scripts/lib/mic/creator.py +1 -1

index f3d0de1..b36bbf0 100644
@@ -18,7 +18,7 @@
 import os, sys, re
 from optparse import SUPPRESS_HELP
 
-from mic import msger, rt_util
+from mic import msger
 from mic.utils import cmdln, errors, rpmmisc
 from mic.conf import configmgr
 from mic.plugin import pluginmgr

scripts/lib/mic/plugins/imager/direct_plugin.py +1 -1

index 53381e5..e015256 100644
@@ -30,7 +30,7 @@ import shutil
 import re
 import tempfile
 
-from mic import chroot, msger, rt_util
+from mic import chroot, msger
 from mic.utils import misc, fs_related, errors, runner, cmdln
 from mic.conf import configmgr
 from mic.plugin import pluginmgr

scripts/lib/mic/plugins/imager/fs_plugin.py +1 -1

index 8e758db..6bcaf00 100644
@@ -18,7 +18,7 @@
 import os
 import sys
 
-from mic import chroot, msger, rt_util
+from mic import chroot, msger
 from mic.utils import cmdln, misc, errors, fs_related
 from mic.imager import fs
 from mic.conf import configmgr

scripts/lib/mic/plugins/imager/livecd_plugin.py +1 -1

index d24ef59..82cb1af 100644
@@ -19,7 +19,7 @@ import os
 import shutil
 import tempfile
 
-from mic import chroot, msger, rt_util
+from mic import chroot, msger
 from mic.utils import misc, fs_related, errors
 from mic.conf import configmgr
 import mic.imager.livecd as livecd

scripts/lib/mic/plugins/imager/liveusb_plugin.py +1 -1

index 7aa8927..3d53c84 100644
@@ -19,7 +19,7 @@ import os
 import shutil
 import tempfile
 
-from mic import chroot, msger, rt_util
+from mic import chroot, msger
 from mic.utils import misc, fs_related, errors
 from mic.utils.partitionedfs import PartitionedMount
 from mic.conf import configmgr

scripts/lib/mic/plugins/imager/loop_plugin.py +1 -1

index 8f4b030..2a05b3c 100644
@@ -19,7 +19,7 @@ import os
 import shutil
 import tempfile
 
-from mic import chroot, msger, rt_util
+from mic import chroot, msger
 from mic.utils import misc, fs_related, errors, cmdln
 from mic.conf import configmgr
 from mic.plugin import pluginmgr

scripts/lib/mic/plugins/imager/raw_plugin.py +1 -1

index 1b9631d..f9625b8 100644
@@ -20,7 +20,7 @@ import shutil
 import re
 import tempfile
 
-from mic import chroot, msger, rt_util
+from mic import chroot, msger
 from mic.utils import misc, fs_related, errors, runner, cmdln
 from mic.conf import configmgr
 from mic.plugin import pluginmgr

Signature · 1 line

1.8.3.1

Tom Zanussi

author

2013-10-22 14:24 UTC #3 lore raw

PATCH 3/10 wic: Remove dependency on rpmmisc

rpmmisc imports rpm and contains misc rpm utilities related to packaging and determining arches based on the packaging. We should never run across this in the initial version of wic, so remove the dependency.

Signed-off-by Tom Zanussi <[email protected]>

---
 scripts/lib/mic/creator.py       | 2 +-
 scripts/lib/mic/imager/livecd.py | 2 +-
 scripts/lib/mic/utils/misc.py    | 1 -
 3 files changed, 2 insertions(+), 3 deletions(-)

scripts/lib/mic/creator.py +1 -1

index b36bbf0..267928f 100644
@@ -19,7 +19,7 @@ import os, sys, re
 from optparse import SUPPRESS_HELP
 
 from mic import msger
-from mic.utils import cmdln, errors, rpmmisc
+from mic.utils import cmdln, errors
 from mic.conf import configmgr
 from mic.plugin import pluginmgr

scripts/lib/mic/imager/livecd.py +1 -1

index a992ee0..0711552 100644
@@ -20,7 +20,7 @@ import glob
 import shutil
 
 from mic import kickstart, msger
-from mic.utils import fs_related, rpmmisc, runner, misc
+from mic.utils import fs_related, runner, misc
 from mic.utils.errors import CreatorError
 from mic.imager.loop import LoopImageCreator

scripts/lib/mic/utils/misc.py +0 -1

index 67ddef2..2c4a3ac 100644
@@ -45,7 +45,6 @@ from mic.utils.fs_related import find_binary_path, makedirs
 from mic.utils.grabber import myurlgrab
 from mic.utils.proxy import get_proxy_for
 from mic.utils import runner
-from mic.utils import rpmmisc
 
 
 RPM_RE  = re.compile("(.*)\.(.*) (.*)-(.*)")

Signature · 1 line

1.8.3.1

Tom Zanussi

author

2013-10-22 14:24 UTC #4 lore raw

PATCH 4/10 wic: Remove dependency on myurlgrab module

In reply to Tom Zanussi

mylrlgrab is in grabber, which imports rpm. For current functionality, we don't need to grab urls or import rpm, so remove the dependency.

Signed-off-by Tom Zanussi <[email protected]>

---
 scripts/lib/mic/utils/misc.py | 1 -
 1 file changed, 1 deletion(-)

scripts/lib/mic/utils/misc.py +0 -1

index 2c4a3ac..95241d7 100644
@@ -42,7 +42,6 @@ xmlparse = cElementTree.parse
 from mic import msger
 from mic.utils.errors import CreatorError, SquashfsError
 from mic.utils.fs_related import find_binary_path, makedirs
-from mic.utils.grabber import myurlgrab
 from mic.utils.proxy import get_proxy_for
 from mic.utils import runner

Signature · 1 line

1.8.3.1

Tom Zanussi

author

2013-10-22 14:24 UTC #5 lore raw

PATCH 5/10 wic: remove rpm warning code from BackendPlugin

In reply to Tom Zanussi

We don't currently use rpm functionality, so we don't need to silence rpm warnings.

Signed-off-by Tom Zanussi <[email protected]>

---
 scripts/lib/mic/pluginbase.py | 5 -----
 1 file changed, 5 deletions(-)

scripts/lib/mic/pluginbase.py +0 -5

index 6ac195b..2f9d720 100644
@@ -83,11 +83,6 @@ class ImagerPlugin(_Plugin):
 class BackendPlugin(_Plugin):
     mic_plugin_type="backend"
 
-    # suppress the verbose rpm warnings
-    if msger.get_loglevel() != 'debug':
-        import rpm
-        rpm.setVerbosity(rpm.RPMLOG_ERR)
-
     def addRepository(self):
         pass

Signature · 1 line

1.8.3.1

Tom Zanussi

author

2013-10-22 14:24 UTC #7 lore raw

PATCH 7/10 wic: Remove rpmmisc call from livecd

In reply to Tom Zanussi

We don't currently use LiveCDImageCreator, but it makes calls when initialized via the plugin interface to rpmmisc module functions, which we don't want the dependency on.

To make it (and LiveUSBImageCreator) happy, we give it the dummy "i386" value for now.

Signed-off-by Tom Zanussi <[email protected]>

---
 scripts/lib/mic/imager/livecd.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

scripts/lib/mic/imager/livecd.py +1 -1

index 0711552..e36f4a7 100644
@@ -741,7 +741,7 @@ hiddenmenu
         self._configure_syslinux_bootloader(isodir)
         self._configure_efi_bootloader(isodir)
 
-arch = rpmmisc.getBaseArch()
+arch = "i386"
 if arch in ("i386", "x86_64"):
     LiveCDImageCreator = x86LiveImageCreator
 elif arch.startswith("arm"):

Signature · 1 line

1.8.3.1

Tom Zanussi

author

2013-10-22 14:24 UTC #9 lore raw

PATCH 9/10 wic: Make find_binary_path() more user-friendly

In reply to Tom Zanussi

find_binary_path() is useful, but if the binary isn't found, it prints a stacktrace and a less-than-useful message. Users complain when they get stacktraces for things they can act on, so remove the stacktrace and tell the user what the problem is.

Signed-off-by Tom Zanussi <[email protected]>

---
 scripts/lib/mic/utils/fs_related.py | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

scripts/lib/mic/utils/fs_related.py +4 -1

index 6161735..dd420e8 100644
@@ -57,7 +57,10 @@ def find_binary_path(binary):
         bin_path = "%s/%s" % (path, binary)
         if os.path.exists(bin_path):
             return bin_path
-    raise CreatorError("Command '%s' is not available." % binary)
+
+    print "External command '%s' not found, exiting." % binary
+    print "  (Please install '%s' on your host system)" % binary
+    sys.exit(1)
 
 def makedirs(dirname):
     """A version of os.makedirs() that doesn't throw an

Signature · 1 line

1.8.3.1

Tom Zanussi

author

2013-10-22 14:24 UTC #10 lore raw

PATCH 10/10 wic: Update and generalize pseudo setup for rootfs generation

In reply to Tom Zanussi

Remove unnecessary pseudo exports i.e. PSEUDO_DISABLED and move the setup to the top-level prepare_rootfs().

Signed-off-by Tom Zanussi <[email protected]>

---
 .../lib/mic/kickstart/custom_commands/partition.py | 28 ++++++++++++----------
 1 file changed, 15 insertions(+), 13 deletions(-)

scripts/lib/mic/kickstart/custom_commands/partition.py +15 ~~-13~~

index 0eb0671..4b11195 100644
@@ -204,25 +204,27 @@ class Wic_PartData(Mic_PartData):
 
         Currently handles ext2/3/4 and btrfs.
         """
+        pseudo = "export PSEUDO_PREFIX=%s/usr;" % native_sysroot
+        pseudo += "export PSEUDO_LOCALSTATEDIR=%s/../pseudo;" % rootfs_dir
+        pseudo += "export PSEUDO_PASSWD=%s;" % rootfs_dir
+        pseudo += "export PSEUDO_NOSYMLINKEXP=1;"
+        pseudo += "%s/usr/bin/pseudo " % native_sysroot
+
         if self.fstype.startswith("ext"):
             return self.prepare_rootfs_ext(cr_workdir, oe_builddir,
-                                           rootfs_dir, native_sysroot)
+                                           rootfs_dir, native_sysroot,
+                                           pseudo)
         elif self.fstype.startswith("btrfs"):
             return self.prepare_rootfs_btrfs(cr_workdir, oe_builddir,
-                                             rootfs_dir, native_sysroot)
+                                             rootfs_dir, native_sysroot,
+                                             pseudo)
 
     def prepare_rootfs_ext(self, cr_workdir, oe_builddir, rootfs_dir,
-                           native_sysroot):
+                           native_sysroot, pseudo):
         """
         Prepare content for an ext2/3/4 rootfs partition.
         """
-        populate_script = "export PSEUDO_PREFIX=%s/usr;" % native_sysroot
-        populate_script += "export PSEUDO_LOCALSTATEDIR=%s/../pseudo;" % rootfs_dir
-        populate_script += "export PSEUDO_PASSWD=%s;" % rootfs_dir
-        populate_script += "export PSEUDO_NOSYMLINKEXP=1;"
-        populate_script += "export PSEUDO_DISABLED=0;"
-        populate_script += "%s/usr/bin/pseudo %s/usr/bin/populate-extfs.sh" % \
-            (native_sysroot, native_sysroot)
+        populate_script = "%s/usr/bin/populate-extfs.sh" % native_sysroot
 
         image_extra_space = 10240

@@ -245,7 +247,7 @@ class Wic_PartData(Mic_PartData):
         rc, out = exec_native_cmd(mkfs_cmd, native_sysroot)
 
         populate_cmd = populate_script + " " + image_rootfs + " " + rootfs
-        rc, out = exec_native_cmd(populate_cmd, native_sysroot)
+        rc, out = exec_native_cmd(pseudo + populate_cmd, native_sysroot)
 
         # get the rootfs size in the right units for kickstart (Mb)
         du_cmd = "du -Lbms %s" % rootfs

@@ -258,7 +260,7 @@ class Wic_PartData(Mic_PartData):
         return 0
 
     def prepare_rootfs_btrfs(self, cr_workdir, oe_builddir, rootfs_dir,
-                             native_sysroot):
+                             native_sysroot, pseudo):
         """
         Prepare content for a btrfs rootfs partition.

@@ -281,7 +283,7 @@ class Wic_PartData(Mic_PartData):
 
         mkfs_cmd = "mkfs.%s -b %d -r %s %s" % \
             (self.fstype, rootfs_size * 1024, image_rootfs, rootfs)
-        rc, out = exec_native_cmd(mkfs_cmd, native_sysroot)
+        rc, out = exec_native_cmd(pseudo + mkfs_cmd, native_sysroot)
 
         # get the rootfs size in the right units for kickstart (Mb)
         du_cmd = "du -Lbms %s" % rootfs

Signature · 1 line

1.8.3.1

Tom Zanussi

author

2013-10-22 14:24 UTC #6 lore raw

PATCH 6/10 wic: Remove rpm and grabber dependencies from BaseImageCreator

In reply to Tom Zanussi

BaseImageCreator is a base class for DirectImageCreator and others, and imports rpm and grabber (which imports rpm).

The various plugins e.g. DirectPlugin import the creators and therefore these dependencies, which manifest at run-time as e.g.:

Warning: Failed to load plugin imager/direct_plugin: No module named
rpm

Signed-off-by Tom Zanussi <[email protected]>

---
 scripts/lib/mic/imager/baseimager.py | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

scripts/lib/mic/imager/baseimager.py +1 -3

index 4d6be29..b721249 100644
@@ -26,12 +26,10 @@ import re
 import tarfile
 import glob
 
-import rpm
-
 from mic import kickstart
 from mic import msger
 from mic.utils.errors import CreatorError, Abort
-from mic.utils import misc, grabber, runner, fs_related as fs
+from mic.utils import misc, runner, fs_related as fs
 
 class BaseImageCreator(object):
     """Installs a system to a chroot directory.

Signature · 1 line

1.8.3.1

Tom Zanussi

author

2013-10-22 14:24 UTC #8 lore raw

PATCH 8/10 wic: Remove binary dependencies

In reply to Tom Zanussi

Current functionality doesn't make use of kpartx, mount, or unmount, and we use native mkswap, so remove the binary checks for those.

Signed-off-by Tom Zanussi <[email protected]>

---
 scripts/lib/mic/utils/partitionedfs.py | 4 ----
 1 file changed, 4 deletions(-)

scripts/lib/mic/utils/partitionedfs.py +0 -4

index 59263ba..6607466 100644
@@ -45,11 +45,7 @@ class PartitionedMount(Mount):
         self.mountOrder = []
         self.unmountOrder = []
         self.parted = find_binary_path("parted")
-        self.kpartx = find_binary_path("kpartx")
-        self.mkswap = find_binary_path("mkswap")
         self.btrfscmd=None
-        self.mountcmd = find_binary_path("mount")
-        self.umountcmd = find_binary_path("umount")
         self.skipformat = skipformat
         self.snapshot_created = self.skipformat
         # Size of a sector used in calculations

Signature · 1 line

1.8.3.1