Skip to contentSkip to conversation

Add initial container IMAGE_FSTYPES support

PATCH In review Randy Witt 2 participants 8 messages Jan 19, 2017 – Jan 30, 2017

lore
Share & tools
lore

Randy Witt

author
#0 lore raw

v2 cover Add initial container IMAGE_FSTYPES support

Updated with Ross' requests.

The following changes since commit a304b94eae99ade6d99800eeaf588c48d01506f6:

oeqa/selftest/devtool: rewrite modify testcase (2017-01-19 22:47:22 +0000)

are available in the git repository at:

git://git.yoctoproject.org/poky-contrib rewitt/contwip
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=rewitt/contwip

Randy Witt (3):
image-container.bbclass: Add the "container" IMAGE_FSTYPES image-container.bbclass: Error if not using linux-dummy selftest/containerimage.py: Add container IMAGE_FSTYPES test

 .../container-image/container-image-testpkg.bb     |  8 +++
 .../container-image/container-test-image.bb        |  8 +++
 meta/classes/image-container.bbclass               | 21 ++++++
 meta/classes/image.bbclass                         |  3 +
 meta/classes/image_types.bbclass                   |  1 +
 meta/lib/oeqa/selftest/containerimage.py           | 82 ++++++++++++++++++++++
 6 files changed, 123 insertions(+)
 create mode 100644 meta-selftest/recipes-test/container-image/container-image-testpkg.bb
 create mode 100644 meta-selftest/recipes-test/container-image/container-test-image.bb
 create mode 100644 meta/classes/image-container.bbclass
 create mode 100644 meta/lib/oeqa/selftest/containerimage.py
Signature · 1 line
2.9.3
author
#1 lore raw

PATCH v2 1/3 image-container.bbclass: Add the "container" IMAGE_FSTYPES

The "container" fstype does very little other than pick tar.bz2 as the actual image type and disable installation of ROOTFS_BOOTSTRAP_INSTALL.

[YOCTO #9502]

Signed-off-by Randy Witt <[email protected]>
---
 meta/classes/image-container.bbclass | 3 +++
 meta/classes/image.bbclass           | 3 +++
 meta/classes/image_types.bbclass     | 1 +
 3 files changed, 7 insertions(+)
 create mode 100644 meta/classes/image-container.bbclass
meta/classes/image-container.bbclass new file +3 -0
new file mode 100644
index 0000000..a10ee32
@@ -0,0 +1,3 @@
+ROOTFS_BOOTSTRAP_INSTALL = ""
+IMAGE_TYPES_MASKED += "container"
+IMAGE_TYPEDEP_container = "tar.bz2"
meta/classes/image.bbclass +3 -0
index 04fd5f9..b2c2a28 100644
@@ -144,6 +144,9 @@ inherit ${IMAGE_TYPE_live}
 IMAGE_TYPE_vm = '${@bb.utils.contains_any("IMAGE_FSTYPES", ["vmdk", "vdi", "qcow2", "hdddirect"], "image-vm", "", d)}'
 inherit ${IMAGE_TYPE_vm}
 
+IMAGE_TYPE_container = '${@bb.utils.contains("IMAGE_FSTYPES", "container", "image-container", "", d)}'
+inherit ${IMAGE_TYPE_container}
+
 def build_uboot(d):
     if 'u-boot' in (d.getVar('IMAGE_FSTYPES') or ''):
         return "image_types_uboot"
meta/classes/image_types.bbclass +1 -0
index 7748dee..727a87e 100644
@@ -328,6 +328,7 @@ IMAGE_TYPES = " \
     hdddirect \
     elf \
     wic wic.gz wic.bz2 wic.lzma \
+    container \
 "
 
 # Compression is a special case of conversion. The old variable
Signature · 1 line
2.9.3
author
#2 lore raw

PATCH v2 2/3 image-container.bbclass: Error if not using linux-dummy

In reply to Randy Witt

In the typical case of a container, the user doesn't particularly care about the kernel version used. Since using linux-dummy prevents building a kernel in the most common case, alert the user if they choose a container image, but don't also specify linux-dummy.

In the cases the user actually does want to build against particular kernel headers, etc, they can override by setting IMAGE_CONTAINER_NO_DUMMY = "1".

[YOCTO #9502]

Signed-off-by Randy Witt <[email protected]>
---
 meta/classes/image-container.bbclass | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)
meta/classes/image-container.bbclass +18 -0
index a10ee32..f002858 100644
@@ -1,3 +1,21 @@
 ROOTFS_BOOTSTRAP_INSTALL = ""
 IMAGE_TYPES_MASKED += "container"
 IMAGE_TYPEDEP_container = "tar.bz2"
+
+python __anonymous() {
+    if "container" in d.getVar("IMAGE_FSTYPES") and \
+       d.getVar("IMAGE_CONTAINER_NO_DUMMY") != "1" and \
+       "linux-dummy" not in d.getVar("PREFERRED_PROVIDER_virtual/kernel"):
+        msg = '"container" is in IMAGE_FSTYPES, but ' \
+              'PREFERRED_PROVIDER_virtual/kernel is not "linux-dummy". ' \
+              'Unless a particular kernel is needed, using linux-dummy will ' \
+              'prevent a kernel from being built, which can reduce ' \
+              'build times. If you don\'t want to use "linux-dummy", set ' \
+              '"IMAGE_CONTAINER_NO_DUMMY" to "1".'
+
+        # Raising skip recipe was Paul's clever idea. It causes the error to
+        # only be shown for the recipes actually requested to build, rather
+        # than bb.fatal which would appear for all recipes inheriting the
+        # class.
+        raise bb.parse.SkipRecipe(msg)
+}
Signature · 1 line
2.9.3
author
#3 lore raw

PATCH v2 3/3 selftest/containerimage.py: Add container IMAGE_FSTYPES test

In reply to Randy Witt

This test checks to make sure only the files expected exist in a container image. Currently only ROOTFS_BOOTSTRAP_INSTALL, gets added to all images without the user specifying it.

But this test should help if a developer in the future ever silently adds more than just ROOTFS_BOOTSTRAP_INSTALL, and that the developer can make sure it also gets removed from a container image.

[YOCTO #9502]

Signed-off-by Randy Witt <[email protected]>
---
 .../container-image/container-image-testpkg.bb     |  8 +++
 .../container-image/container-test-image.bb        |  8 +++
 meta/lib/oeqa/selftest/containerimage.py           | 82 ++++++++++++++++++++++
 3 files changed, 98 insertions(+)
 create mode 100644 meta-selftest/recipes-test/container-image/container-image-testpkg.bb
 create mode 100644 meta-selftest/recipes-test/container-image/container-test-image.bb
 create mode 100644 meta/lib/oeqa/selftest/containerimage.py
meta-selftest/recipes-test/container-image/container-image-testpkg.bb new file +8 -0
new file mode 100644
index 0000000..f8dd229
@@ -0,0 +1,8 @@
+LICENSE = "MIT"
+
+INHIBIT_DEFAULT_DEPS = "1"
+
+do_install_append() {
+    install -d ${D}${bindir}
+    touch ${D}${bindir}/theapp
+}
meta-selftest/recipes-test/container-image/container-test-image.bb new file +8 -0
new file mode 100644
index 0000000..d5f939c
@@ -0,0 +1,8 @@
+IMAGE_INSTALL += "container-image-testpkg"
+
+LICENSE = "MIT"
+
+IMAGE_FSTYPES = "container"
+IMAGE_LINGUAS = ""
+
+inherit core-image
meta/lib/oeqa/selftest/containerimage.py new file +82 -0

82 lines not on this page · raw message

Signature · 1 line
2.9.3
#4 lore raw

Re: [PATCH v2 3/3] selftest/containerimage.py: Add container IMAGE_FSTYPES test

The test fetches bitbake variables and then changes the configuration, so it fails if the user's MACHINE isn't qemux86. I've changed this locally and now its failing differently:

First extra element 18:
./etc/systemd/system/sysinit.target.wants/opkg-configure.service

Looks like you want to lock the configuration to sysvinit.

Ross

Quoted 160 lines · Randy Witt

This test checks to make sure only the files expected exist in a container image. Currently only ROOTFS_BOOTSTRAP_INSTALL, gets added to all images without the user specifying it.

But this test should help if a developer in the future ever silently adds more than just ROOTFS_BOOTSTRAP_INSTALL, and that the developer can make sure it also gets removed from a container image.

[YOCTO #9502]

Signed-off-by Randy Witt <[email protected]>
---
 .../container-image/container-image-testpkg.bb     |  8 +++
 .../container-image/container-test-image.bb        |  8 +++
 meta/lib/oeqa/selftest/containerimage.py           | 82
++++++++++++++++++++++
 3 files changed, 98 insertions(+)
 create mode 100644 meta-selftest/recipes-test/container-image/container-
image-testpkg.bb
 create mode 100644 meta-selftest/recipes-test/container-image/container-
test-image.bb
 create mode 100644 meta/lib/oeqa/selftest/containerimage.py
(quoted diff) +0 -0

image-testpkg.bb b/meta-selftest/recipes-test/container-image/container- image-testpkg.bb
new file mode 100644
index 0000000..f8dd229

meta-selftest/recipes-test/container-image/container- new file +0 -0

image-testpkg.bb

meta-selftest/recipes-test/container-image/container- new file +8 -0
@@ -0,0 +1,8 @@
+LICENSE = "MIT"
+
+INHIBIT_DEFAULT_DEPS = "1"
+
+do_install_append() {
+    install -d ${D}${bindir}
+    touch ${D}${bindir}/theapp
+}
(quoted diff) +0 -0

test-image.bb b/meta-selftest/recipes-test/container-image/container- test-image.bb
new file mode 100644
index 0000000..d5f939c

meta-selftest/recipes-test/container-image/container-test-image.bb new file +8 -0
@@ -0,0 +1,8 @@
+IMAGE_INSTALL += "container-image-testpkg"
+
+LICENSE = "MIT"
+
+IMAGE_FSTYPES = "container"
+IMAGE_LINGUAS = ""
+
+inherit core-image
meta/lib/oeqa/selftest/containerimage.py new file +10 -0
new file mode 100644
index 0000000..6a85222
@@ -0,0 +1,82 @@
+import os
+
+from oeqa.selftest.base import oeSelfTest
+from oeqa.utils.commands import bitbake, get_bb_vars, runCmd
+
+# This test builds an image with using the "container" IMAGE_FSTYPE, and
+# ensures that then files in the image are only the ones expected.
+#
+# The only package added to the image is container_image_testpkg, which
+# contains one file. However, due to some other things not cleaning up

during

meta/lib/oeqa/selftest/containerimage.py new file +4 -0
+# rootfs creation, there is some cruft. Ideally bugs will be filed and the
+# cruft removed, but for now we whitelist some known set.
+#
+# Also for performance reasons we're only checking the cruft when using

ipk.

meta/lib/oeqa/selftest/containerimage.py new file +7 -0
+# When using deb, and rpm it is a bit different and we could test all
+# of them, but this test is more to catch if other packages get added by
+# default other than what is in ROOTFS_BOOTSTRAP_INSTALL.
+#
+class ContainerImageTests(oeSelfTest):
+
+    # Verify that when specifying a IMAGE_TYPEDEP_ of the form "foo.bar"

that

meta/lib/oeqa/selftest/containerimage.py new file +36 -0
+    # the conversion type bar gets added as a dep as well
+    def test_expected_files(self):
+
+        def get_each_path_part(path):
+            if path:
+                part = [ '.' + path + '/' ]
+                result = get_each_path_part(path.rsplit('/', 1)[0])
+                if result:
+                    return part + result
+                else:
+                    return part
+            else:
+                return None
+
+        bbvars = get_bb_vars(['bindir', 'sysconfdir', 'localstatedir',
+                              'DEPLOY_DIR_IMAGE'])
+        expected_files = [
+                    './',
+                    '.{bindir}/theapp',
+                    '.{sysconfdir}/default/',
+                    '.{sysconfdir}/default/postinst',
+                    '.{sysconfdir}/ld.so.cache',
+                    '.{sysconfdir}/timestamp',
+                    '.{sysconfdir}/version',
+                    './run/',
+                    '.{localstatedir}/cache/',
+                    '.{localstatedir}/cache/ldconfig/',
+                    '.{localstatedir}/cache/ldconfig/aux-cache',
+                    '.{localstatedir}/cache/opkg/',
+                    '.{localstatedir}/lib/',
+                    '.{localstatedir}/lib/opkg/'
+                ]
+
+        expected_files = [ x.format(bindir=bbvars['bindir'],
+                                    sysconfdir=bbvars['sysconfdir'],
+                                    localstatedir=bbvars['

localstatedir'])

meta/lib/oeqa/selftest/containerimage.py new file +3 -0
+                                    for x in expected_files ]
+
+        # Since tar lists all directories individually, make sure each

element

meta/lib/oeqa/selftest/containerimage.py new file +9 -0
+        # from bindir, sysconfdir, etc is added
+        expected_files += get_each_path_part(bbvars['bindir'])
+        expected_files += get_each_path_part(bbvars['sysconfdir'])
+        expected_files += get_each_path_part(bbvars['localstatedir'])
+
+        expected_files = sorted(expected_files)
+
+
+        self.write_config('PREFERRED_PROVIDER_virtual/kernel =

"linux-dummy"')

meta/lib/oeqa/selftest/containerimage.py new file +13 -1
+        self.append_config('MACHINE = "qemux86"')
+        self.append_config('IMAGE_FSTYPES = "container"')
+        self.append_config('PACKAGE_CLASSES = "package_ipk"')
+
+        # Build the image of course
+        bitbake('container-test-image')
+
+        image = os.path.join(bbvars['DEPLOY_DIR_IMAGE'],
+                             'container-test-image-qemux86.tar.bz2')
+
+        # Ensure the files in the image are what we expect
+        result = runCmd("tar tf {} | sort".format(image), shell=True)
+        self.assertEqual(result.output.split('\n'), expected_files)
--

2.9.3

--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

author
#5 lore raw

PATCH v3 3/3 selftest/containerimage.py: Add container IMAGE_FSTYPES test

In reply to Randy Witt

This test checks to make sure only the files expected exist in a container image. Currently only ROOTFS_BOOTSTRAP_INSTALL, gets added to all images without the user specifying it.

But this test should help if a developer in the future ever silently adds more than just ROOTFS_BOOTSTRAP_INSTALL, and that the developer can make sure it also gets removed from a container image.

[YOCTO #9502]

Signed-off-by Randy Witt <[email protected]>
---
 .../container-image/container-image-testpkg.bb     |  8 +++
 .../container-image/container-test-image.bb        |  8 +++
 meta/lib/oeqa/selftest/containerimage.py           | 82 ++++++++++++++++++++++
 3 files changed, 98 insertions(+)
 create mode 100644 meta-selftest/recipes-test/container-image/container-image-testpkg.bb
 create mode 100644 meta-selftest/recipes-test/container-image/container-test-image.bb
 create mode 100644 meta/lib/oeqa/selftest/containerimage.py
meta-selftest/recipes-test/container-image/container-image-testpkg.bb new file +8 -0
new file mode 100644
index 0000000..f8dd229
@@ -0,0 +1,8 @@
+LICENSE = "MIT"
+
+INHIBIT_DEFAULT_DEPS = "1"
+
+do_install_append() {
+    install -d ${D}${bindir}
+    touch ${D}${bindir}/theapp
+}
meta-selftest/recipes-test/container-image/container-test-image.bb new file +8 -0
new file mode 100644
index 0000000..d5f939c
@@ -0,0 +1,8 @@
+IMAGE_INSTALL += "container-image-testpkg"
+
+LICENSE = "MIT"
+
+IMAGE_FSTYPES = "container"
+IMAGE_LINGUAS = ""
+
+inherit core-image
meta/lib/oeqa/selftest/containerimage.py new file +82 -0

82 lines not on this page · raw message

Signature · 1 line
2.9.3
#6 lore raw

Re: [PATCH v3 3/3] selftest/containerimage.py: Add container IMAGE_FSTYPES test

FAIL [40.619s]: test_expected_files
(oeqa.selftest.containerimage.ContainerImageTests)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/lib/oeqa/selftest/containerimage.py", line 82, in test_expected_files
self.assertEqual(result.output.split('\n'), expected_files)
AssertionError: Lists differ: ['tar:
/home/pokybuild/yocto-autobuilder/yo[191 chars]now'] != ['./', './etc/',
'./etc/default/', './etc/d[262 chars]kg/']

First differing element 0:
tar:
/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/deploy/images/qemux86-64/container-test-image-qemux86.tar.bz2:
Cannot open: No such file or directory
./

Second list contains 16 additional elements.
First extra element 2:
./etc/default/

- ['tar: '
-
'/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/build/tmp/deploy/images/qemux86-64/container-test-image-qemux86.tar.bz2:
'
- 'Cannot open: No such file or directory',
- 'tar: Error is not recoverable: exiting now']
+ ['./',

+  './etc/',
+  './etc/default/',
+  './etc/default/postinst',
+  './etc/ld.so.cache',
+  './etc/timestamp',
+  './etc/version',
+  './run/',
+  './usr/',
+  './usr/bin/',
+  './usr/bin/theapp',
+  './var/',
+  './var/cache/',
+  './var/cache/ldconfig/',
+  './var/cache/ldconfig/aux-cache',
+  './var/cache/opkg/',
+  './var/lib/',
+  './var/lib/opkg/']

The test needs to handle tar failing, and appears to be confused as to whether its building for qemux86 or qemux86-64 (see the path that tar tries to list).

Ross

author
#7 lore raw

PATCH v4 3/3 selftest/containerimage.py: Add container IMAGE_FSTYPES test

In reply to Randy Witt

This test checks to make sure only the files expected exist in a container image. Currently only ROOTFS_BOOTSTRAP_INSTALL, gets added to all images without the user specifying it.

But this test should help if a developer in the future ever silently adds more than just ROOTFS_BOOTSTRAP_INSTALL, and that the developer can make sure it also gets removed from a container image.

[YOCTO #9502]

Signed-off-by Randy Witt <[email protected]>
---
 .../container-image/container-image-testpkg.bb     |  8 +++
 .../container-image/container-test-image.bb        |  8 +++
 meta/lib/oeqa/selftest/containerimage.py           | 83 ++++++++++++++++++++++
 3 files changed, 99 insertions(+)
 create mode 100644 meta-selftest/recipes-test/container-image/container-image-testpkg.bb
 create mode 100644 meta-selftest/recipes-test/container-image/container-test-image.bb
 create mode 100644 meta/lib/oeqa/selftest/containerimage.py
meta-selftest/recipes-test/container-image/container-image-testpkg.bb new file +8 -0
new file mode 100644
index 0000000..f8dd229
@@ -0,0 +1,8 @@
+LICENSE = "MIT"
+
+INHIBIT_DEFAULT_DEPS = "1"
+
+do_install_append() {
+    install -d ${D}${bindir}
+    touch ${D}${bindir}/theapp
+}
meta-selftest/recipes-test/container-image/container-test-image.bb new file +8 -0
new file mode 100644
index 0000000..d5f939c
@@ -0,0 +1,8 @@
+IMAGE_INSTALL += "container-image-testpkg"
+
+LICENSE = "MIT"
+
+IMAGE_FSTYPES = "container"
+IMAGE_LINGUAS = ""
+
+inherit core-image
meta/lib/oeqa/selftest/containerimage.py new file +83 -0

83 lines not on this page · raw message

Signature · 1 line
2.9.3