Skip to contentSkip to conversation

#11662 - wic should mount /boot

PATCH Superseded Ed Bartosh 1 participant 7 messages Jul 27, 2017

v4 of 5 Superseded Latest is v5 →

lore
Share & tools
lore

Ed Bartosh

author
#0 lore raw

v4 cover #11662 - wic should mount /boot

Hi,

This patchset adds /boot to the /etc/fstab of root partition, making it mounted on boot. It also fixes reporting and testing issues caused by this change.

The patchset also fixes long standing bug: wic updated fstab inplace in rootfs directory. This causes other tasks working with rootfs directory to produce incorrect results or crash. This is fixed by copying rootfs content to the temporary directory before updating fstab.

In previous versions of this patchset temporary rootfs directory was created using copyhardlinktree API. This broke do_image_tar as creating hardlinks changes file ctime and causes tar to exit with the error "file changed as we read it". It's fixes in v4 by using copying instead of hardlinking. This is slower, but it shouldn't influence overall build time too much as even for a rootfs size 500Mb it takes only 1.5 sec

Changes in v2: squashed patches by reviewer's request Changes in v3: unlink /etc/fstab in rootfs copy before updating it Changes in v4: used 'cp -a' instead of copyhardlinktree to avoid
do_image_tar failure due to changed ctime

The following changes since commit b73f5e088a543775a2a94b60302f750edfffbd10:

wic-tools: add dependency to e2fsprogs-native (2017-07-27 16:07:26 +0300)

are available in the git repository at:

git://git.yoctoproject.org/poky-contrib ed/wip
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/wip

Ed Bartosh (6):
wic: copy rootfs directory before changing fstab
wic: use absolute paths in rootfs plugin
wic: rootfs: fix rootfs path reporting
wic: rootfs: make copied rootfs unique
wic: add /boot mount point to fstab by default
oe-selftest: wic: fix test_quemu

 meta/lib/oeqa/selftest/cases/wic.py      |  2 +-
 scripts/lib/wic/plugins/imager/direct.py | 24 ++++++++++++++----------
 scripts/lib/wic/plugins/source/rootfs.py | 16 +++++++---------
 3 files changed, 22 insertions(+), 20 deletions(-)

--
Regards,
Ed

author
#1 lore raw

PATCH v4 1/6 wic: copy rootfs directory before changing fstab

wic updates /etc/fstab on root partition if there are valid mount points in .wks

When wic runs from bitbake this can cause incorrect results or even breakage of other tasks working with the same rootfs directory in parallel with do_image_wic.

Implemented copying rootfs directory to a temporary location before updating fstab to avoid conflicts with other tasks working with the same rootfs directory.

---
 scripts/lib/wic/plugins/imager/direct.py | 22 +++++++++++++---------
 1 file changed, 13 insertions(+), 9 deletions(-)
scripts/lib/wic/plugins/imager/direct.py +13 -9
index f20d843..07e3f3e 100644
@@ -115,12 +115,18 @@ class DirectPlugin(ImagerPlugin):
             fstab_lines = fstab.readlines()
 
         if self._update_fstab(fstab_lines, self.parts):
-            shutil.copyfile(fstab_path, fstab_path + ".orig")
+            # copy rootfs dir to workdir to update fstab
+            # as rootfs can be used by other tasks and can't be modified
+            new_rootfs = os.path.realpath(os.path.join(self.workdir, "rootfs_copy"))
+            exec_cmd("cp -a %s %s" % (image_rootfs, new_rootfs),
+                                      self.native_sysroot)
+
+            fstab_path = os.path.join(new_rootfs, 'etc/fstab')
 
             with open(fstab_path, "w") as fstab:
                 fstab.writelines(fstab_lines)
 
-            return fstab_path
+            return new_rootfs
 
     def _update_fstab(self, fstab_lines, parts):
         """Assume partition order same as in wks"""
@@ -156,7 +162,10 @@ class DirectPlugin(ImagerPlugin):
         filesystems from the artifacts directly and combine them into
         a partitioned image.
         """
-        fstab_path = self._write_fstab(self.rootfs_dir.get("ROOTFS_DIR"))
+        new_rootfs = self._write_fstab(self.rootfs_dir.get("ROOTFS_DIR"))
+        if new_rootfs:
+            # rootfs was copied to update fstab
+            self.rootfs_dir['ROOTFS_DIR'] = new_rootfs
 
         for part in self.parts:
             # get rootfs size from bitbake variable if it's not set in .ks file
@@ -172,12 +181,7 @@ class DirectPlugin(ImagerPlugin):
                     if rsize_bb:
                         part.size = int(round(float(rsize_bb)))
 
-        try:
-            self._image.prepare(self)
-        finally:
-            if fstab_path:
-                shutil.move(fstab_path + ".orig", fstab_path)
-
+        self._image.prepare(self)
         self._image.layout_partitions()
         self._image.create()
 
Signature · 1 line
2.1.4
author
#2 lore raw

PATCH v4 2/6 wic: use absolute paths in rootfs plugin

In reply to Ed Bartosh

Using relative paths can cause copyhardlinktree API to fail as it changes current directory when working. Converted all paths to absolute paths using os.path.realpath.

---
 scripts/lib/wic/plugins/source/rootfs.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
scripts/lib/wic/plugins/source/rootfs.py +2 -2
index 4dc4cb8..c08f760 100644
@@ -48,7 +48,7 @@ class RootfsPlugin(SourcePlugin):
     @staticmethod
     def __get_rootfs_dir(rootfs_dir):
         if os.path.isdir(rootfs_dir):
-            return rootfs_dir
+            return os.path.realpath(rootfs_dir)
 
         image_rootfs_dir = get_bitbake_var("IMAGE_ROOTFS", rootfs_dir)
         if not os.path.isdir(image_rootfs_dir):
@@ -56,7 +56,7 @@ class RootfsPlugin(SourcePlugin):
                            "named %s has been found at %s, exiting." %
                            (rootfs_dir, image_rootfs_dir))
 
-        return image_rootfs_dir
+        return os.path.realpath(image_rootfs_dir)
 
     @classmethod
     def do_prepare_partition(cls, part, source_params, cr, cr_workdir,
Signature · 1 line
2.1.4
author
#3 lore raw

PATCH v4 3/6 wic: rootfs: fix rootfs path reporting

In reply to Ed Bartosh

wic gets rootfs paths from partition object property 'rootfs_dir' and shows them in final report.

rootfs plugin sets this property to the temporary path, which causes temporary paths appearing in the report.

Changed the code to prevent storing temporary rootfs path in part.rootfs_dir. This should fix the report.

---
 scripts/lib/wic/plugins/source/rootfs.py | 10 ++++------
 1 file changed, 4 insertions(+), 6 deletions(-)
scripts/lib/wic/plugins/source/rootfs.py +4 -6
index c08f760..e438158 100644
@@ -81,8 +81,9 @@ class RootfsPlugin(SourcePlugin):
                 raise WicError("Couldn't find --rootfs-dir=%s connection or "
                                "it is not a valid path, exiting" % part.rootfs_dir)
 
-        real_rootfs_dir = cls.__get_rootfs_dir(rootfs_dir)
+        part.rootfs_dir = cls.__get_rootfs_dir(rootfs_dir)
 
+        new_rootfs = None
         # Handle excluded paths.
         if part.exclude_path is not None:
             # We need a new rootfs directory we can delete files from. Copy to
@@ -92,9 +93,7 @@ class RootfsPlugin(SourcePlugin):
             if os.path.lexists(new_rootfs):
                 shutil.rmtree(os.path.join(new_rootfs))
 
-            copyhardlinktree(real_rootfs_dir, new_rootfs)
-
-            real_rootfs_dir = new_rootfs
+            copyhardlinktree(part.rootfs_dir, new_rootfs)
 
             for orig_path in part.exclude_path:
                 path = orig_path
@@ -123,6 +122,5 @@ class RootfsPlugin(SourcePlugin):
                     # Delete whole directory.
                     shutil.rmtree(full_path)
 
-        part.rootfs_dir = real_rootfs_dir
         part.prepare_rootfs(cr_workdir, oe_builddir,
-                            real_rootfs_dir, native_sysroot)
+                            new_rootfs or part.rootfs_dir, native_sysroot)
Signature · 1 line
2.1.4
author
#4 lore raw

PATCH v4 4/6 wic: rootfs: make copied rootfs unique

In reply to Ed Bartosh

Used unique suffix (line number from .wks file) for the copied rootfs directory to avoid possible conflicts.

---
 scripts/lib/wic/plugins/source/rootfs.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/lib/wic/plugins/source/rootfs.py +1 -1
index e438158..aec720f 100644
@@ -88,7 +88,7 @@ class RootfsPlugin(SourcePlugin):
         if part.exclude_path is not None:
             # We need a new rootfs directory we can delete files from. Copy to
             # workdir.
-            new_rootfs = os.path.realpath(os.path.join(cr_workdir, "rootfs"))
+            new_rootfs = os.path.realpath(os.path.join(cr_workdir, "rootfs%d" % part.lineno))
 
             if os.path.lexists(new_rootfs):
                 shutil.rmtree(os.path.join(new_rootfs))
Signature · 1 line
2.1.4
author
#5 lore raw

PATCH v4 5/6 wic: add /boot mount point to fstab by default

In reply to Ed Bartosh

wic avoided adding /boot to fstab for no reason.
This exception was hardcoded in the wic code.

There is no need for this as mountpoint in .wks file is an optional field. It can be used only if user wants to have partitions automatically mounted on system boot.

[YOCTO #11662]

---
 scripts/lib/wic/plugins/imager/direct.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/lib/wic/plugins/imager/direct.py +1 -1
index 07e3f3e..61500ab 100644
@@ -133,7 +133,7 @@ class DirectPlugin(ImagerPlugin):
         updated = False
         for part in parts:
             if not part.realnum or not part.mountpoint \
-               or part.mountpoint in ("/", "/boot"):
+               or part.mountpoint == "/":
                 continue
 
             # mmc device partitions are named mmcblk0p1, mmcblk0p2..
Signature · 1 line
2.1.4
author
#6 lore raw

PATCH v4 6/6 oe-selftest: wic: fix test_quemu

In reply to Ed Bartosh

This test case boots the image in qemu and checks for mounted partitions. As /boot is mounted automatically the test case fails. Fixed this by adding /boot to the list of mounted partitions.

---
 meta/lib/oeqa/selftest/cases/wic.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
meta/lib/oeqa/selftest/cases/wic.py +1 -1
index 7c2c877..aaefd4f 100644
@@ -637,7 +637,7 @@ part /etc --source rootfs --ondisk mmcblk0 --fstype=ext4 --exclude-path bin/ --r
             cmd = "mount |grep '^/dev/' | cut -f1,3 -d ' '"
             status, output = qemu.run_serial(cmd)
             self.assertEqual(1, status, 'Failed to run command "%s": %s' % (cmd, output))
-            self.assertEqual(output, '/dev/root /\r\n/dev/sda3 /mnt')
+            self.assertEqual(output, '/dev/root /\r\n/dev/sda1 /boot\r\n/dev/sda3 /mnt')
 
     @only_for_arch(['i586', 'i686', 'x86_64'])
     @OETestID(1852)
Signature · 1 line
2.1.4