This is an implementation of new wic subcommand 'wic cp'.
cover This is an implementation of new wic subcommand 'wic cp'.
Hi,
'wic cp' command copies files and directories to the vfat partition of the wic image.
Here are couple of examples:
Copy file to the root directory of the vfat partition:
$ wic cp test.wks tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.wic:1
$ wic ls tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.wic:1
Volume in drive : is boot
Volume Serial Number is DB4C-FD4C
Directory for ::/ efi <DIR> 2017-05-24 18:15
loader <DIR> 2017-05-24 18:15
startup nsh 26 2017-05-24 18:15
vmlinuz 6926384 2017-05-24 18:15
test wks 628 2017-05-24 21:22 5 files 6 927 038 bytes
15 677 440 bytes free
Copy directory to the efi subdirectory of the vfat partition:
$ wic cp test tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.wic:1/efi/
$ wic ls tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.wic:1/eti/
Volume in drive : is boot
Volume Serial Number is DB4C-FD4C
Directory for ::/efi . <DIR> 2017-05-24 18:15
.. <DIR> 2017-05-24 18:15
boot <DIR> 2017-05-24 18:15
test <DIR> 2017-05-24 21:27 4 files 0 bytes
15 675 392 bytes free
The patchset also contains fixes for sparse_copy bugs discuvered in the process of working on 'wic cp' functionality.
It also contains test case for 'wic cp' command.
The following changes since commit ae4b01e75618030f3fa0a592c6dffddb2fdab3b5:
selftest: add new test case test_wic_ls (2017-05-25 17:30:03 +0300)
are available in the git repository at:
git://git.yoctoproject.org/poky-contrib ed/wic/wip
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/wic/wip
Ed Bartosh (11):
wic: add wic_init_parser_cp
wic: add help and usage content for 'wic cp'
wic: add 'wic cp' command
wic: add Disk._prop helper
wic: add mcopy property
filemap: change signature of sparse_copy function
filemap: check if dest is written for every block
filemap: calculate dst size correctly
wic: add Disk._put_part_image method
wic: fully implement 'wic cp'
selftest: add test_wic_cp test case
meta/lib/oeqa/selftest/wic.py | 46 +++++++++++++++++++++++- scripts/lib/wic/engine.py | 41 ++++++++++++++++++--- scripts/lib/wic/filemap.py | 28 ++++++++++++--- scripts/lib/wic/help.py | 62 ++++++++++++++++++++++++++++++++ scripts/lib/wic/plugins/imager/direct.py | 2 +- scripts/wic | 29 +++++++++++++++ 6 files changed, 197 insertions(+), 11 deletions(-)
--
Regards,
Ed
PATCH 1/11 wic: add wic_init_parser_cp
Add parser for 'wic cp' subcommand and a custom argument type.
--- scripts/wic | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)
scripts/wic +16 -0
@@ -341,6 +341,8 @@ def imgtype(arg): image, part = image.split(':') if '/' in part: part, path = part.split('/', 1) + if not path: + path = '/' if not os.path.isfile(image): err = "%s is not a regular file or symlink" % image
@@ -354,6 +356,20 @@ def wic_init_parser_ls(subparser): subparser.add_argument("-n", "--native-sysroot", help="path to the native sysroot containing the tools") +def imgpathtype(arg): + img = imgtype(arg) + if img.part is None: + raise argparse.ArgumentTypeError("partition number is not specified") + return img + +def wic_init_parser_cp(subparser): + subparser.add_argument("src", + help="source spec") + subparser.add_argument("dest", type=imgpathtype, + help="image spec: <image>:<vfat partition>[<path>]") + subparser.add_argument("-n", "--native-sysroot", + help="path to the native sysroot containing the tools") + def wic_init_parser_help(subparser): helpparsers = subparser.add_subparsers(dest='help_topic', help=hlp.wic_usage) for helptopic in helptopics:
Signature · 1 line
2.1.4
PATCH 2/11 wic: add help and usage content for 'wic cp'
In reply to Ed BartoshAdded wic_cp_help and wic_cp_usage variables to help.py. These variables contain help content that will be used in 'wic cp help' and 'wic cp --help' output.
[YOCTO #11283]
--- scripts/lib/wic/help.py | 62 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 62 insertions(+)
scripts/lib/wic/help.py +62 -0
@@ -349,6 +349,68 @@ DESCRIPTION """ +wic_cp_usage = """ + + Copy files and directories to the vfat partitions + + usage: wic cp <src> <image>:<vfat partition>[<path>] [--native-sysroot <path>] + + This command copies local files or directories to the vfat partitions of partitioned + image. + + See 'wic help cp' for more detailed instructions. + +""" + +wic_cp_help = """ + +NAME + wic cp - copy files and directories to the vfat partitions + +SYNOPSIS + wic cp <src> <image>:<vfat partition> + wic cp <src> <image>:<vfat partition><path> + wic cp <src> <image>:<vfat partition><path> --native-sysroot <path> + +DESCRIPTION + This command copies files and directories to the vfat partition of the + wic image. + + The first form of it copies file or directory to the root directory of + the vfat partition: + $ wic cp test.wks tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.wic:1 + $ wic ls tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.wic:1 + Volume in drive : is boot + Volume Serial Number is DB4C-FD4C + Directory for ::/ + + efi <DIR> 2017-05-24 18:15 + loader <DIR> 2017-05-24 18:15 + startup nsh 26 2017-05-24 18:15 + vmlinuz 6926384 2017-05-24 18:15 + test wks 628 2017-05-24 21:22 + 5 files 6 927 038 bytes + 15 677 440 bytes free + + The second form of the command copies file or directory to the specified directory + on the vfat partition: + $ wic cp test tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.wic:1/efi/ + $ wic ls tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64.wic:1/eti/ + Volume in drive : is boot + Volume Serial Number is DB4C-FD4C + Directory for ::/efi + + . <DIR> 2017-05-24 18:15 + .. <DIR> 2017-05-24 18:15 + boot <DIR> 2017-05-24 18:15 + test <DIR> 2017-05-24 21:27 + 4 files 0 bytes + 15 675 392 bytes free + + The -n option is used to specify the path to the native sysroot + containing the tools(parted and mtools) to use. +""" + wic_plugins_help = """ NAME
Signature · 1 line
2.1.4
PATCH 3/11 wic: add 'wic cp' command
In reply to Ed BartoshAdded empty 'wic cp' command that does nothing. The functionality will be added by the next commits.
[YOCTO #11283]
--- scripts/lib/wic/engine.py | 7 +++++++ scripts/wic | 13 +++++++++++++ 2 files changed, 20 insertions(+)
scripts/lib/wic/engine.py +7 -0
@@ -309,6 +309,13 @@ def wic_ls(args, native_sysroot): path = args.path.path or '/' print(disk.dir(args.path.part, path)) +def wic_cp(args, native_sysroot): + """ + Copy local file or directory to the vfat partition of + partitioned image. + """ + pass + def find_canned(scripts_path, file_name): """ Find a file either by its path or by name in the canned files dir.
scripts/wic +13 -0
@@ -241,6 +241,12 @@ def wic_ls_subcommand(args, usage_str): """ engine.wic_ls(args, args.native_sysroot) +def wic_cp_subcommand(args, usage_str): + """ + Command-line handling for copying files/dirs to images. + The real work is done by engine.wic_cp() + """ + engine.wic_cp(args, args.native_sysroot) def wic_help_subcommand(args, usage_str): """
@@ -277,6 +283,9 @@ helptopics = { "ls": [wic_help_topic_subcommand, wic_help_topic_usage, hlp.wic_ls_help], + "cp": [wic_help_topic_subcommand, + wic_help_topic_usage, + hlp.wic_cp_help], "list": [wic_help_topic_subcommand, wic_help_topic_usage, hlp.wic_list_help]
@@ -390,6 +399,10 @@ subcommands = { hlp.wic_ls_usage, hlp.wic_ls_help, wic_init_parser_ls], + "cp": [wic_cp_subcommand, + hlp.wic_cp_usage, + hlp.wic_cp_help, + wic_init_parser_cp], "help": [wic_help_subcommand, wic_help_topic_usage, hlp.wic_help_help,
Signature · 1 line
2.1.4
PATCH 5/11 wic: add mcopy property
In reply to Ed BartoshAdded property that points to the mcopy executable.
--- scripts/lib/wic/engine.py | 5 +++++ 1 file changed, 5 insertions(+)
scripts/lib/wic/engine.py +5 -0
@@ -237,6 +237,7 @@ class Disk: self.native_sysroot = native_sysroot self._partitions = None self._mdir = None + self._mcopy = None self._partimages = {} # find parted
@@ -280,6 +281,10 @@ class Disk: def mdir(self): return self._prop('mdir') + @property + def mcopy(self): + return self._prop("mcopy") + def _get_part_image(self, pnum): if pnum not in self.partitions: raise WicError("Partition %s is not in the image")
Signature · 1 line
2.1.4
PATCH 4/11 wic: add Disk._prop helper
In reply to Ed BartoshAdded generic helper to use in property methods to access commands in a lazy manner.
--- scripts/lib/wic/engine.py | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-)
scripts/lib/wic/engine.py +10 -5
@@ -267,13 +267,18 @@ class Disk: return self._partitions + def _prop(self, name): + """Get path to the executable in a lazy way.""" + aname = "_%s" % name + if getattr(self, aname) is None: + setattr(self, aname, find_executable(name, self.paths)) + if not getattr(self, aname): + raise WicError("Can't find executable {}".format(name)) + return getattr(self, aname) + @property def mdir(self): - if self._mdir is None: - self._mdir = find_executable("mdir", self.paths) - if not self._mdir: - raise WicError("Can't find executable mdir") - return self._mdir + return self._prop('mdir') def _get_part_image(self, pnum): if pnum not in self.partitions:
Signature · 1 line
2.1.4
PATCH 6/11 filemap: change signature of sparse_copy function
In reply to Ed BartoshRenamed parameter offset->skip to match names of dd parameters.
Changed affected sparse_copy calls.
Added explanation of the parameters to docstring.
--- scripts/lib/wic/filemap.py | 15 ++++++++++++--- scripts/lib/wic/plugins/imager/direct.py | 2 +- 2 files changed, 13 insertions(+), 4 deletions(-)
scripts/lib/wic/filemap.py +12 -3
@@ -530,9 +530,18 @@ def filemap(image, log=None): except ErrorNotSupp: return FilemapSeek(image, log) -def sparse_copy(src_fname, dst_fname, offset=0, skip=0, +def sparse_copy(src_fname, dst_fname, skip=0, seek=0, length=0, api=None): - """Efficiently copy sparse file to or into another file.""" + """ + Efficiently copy sparse file to or into another file. + + src_fname: path to source file + dst_fname: path to destination file + skip: skip N bytes at thestart of src + seek: seek N bytes from the start of dst + length: read N bytes from src and write them to dst + api: FilemapFiemap or FilemapSeek object + """ if not api: api = filemap fmap = api(src_fname)
@@ -554,7 +563,7 @@ def sparse_copy(src_fname, dst_fname, offset=0, skip=0, start = skip fmap._f_image.seek(start, os.SEEK_SET) - dst_file.seek(offset + start - skip, os.SEEK_SET) + dst_file.seek(seek + start - skip, os.SEEK_SET) chunk_size = 1024 * 1024 to_read = end - start
scripts/lib/wic/plugins/imager/direct.py +1 -1
@@ -550,7 +550,7 @@ class PartitionedImage(): source = part.source_file if source: # install source_file contents into a partition - sparse_copy(source, self.path, part.start * self.sector_size) + sparse_copy(source, self.path, seek=part.start * self.sector_size) logger.debug("Installed %s in partition %d, sectors %d-%d, " "size %d sectors", source, part.num, part.start,
Signature · 1 line
2.1.4
PATCH 7/11 filemap: check if dest is written for every block
In reply to Ed BartoshIf lenght parameter is provided to sparse_copy call it's mandatory to check if the output file is fully written after reading unmapped block from input file.
If it's not done then sparse_copy can write more data than specified length.
--- scripts/lib/wic/filemap.py | 7 +++++++ 1 file changed, 7 insertions(+)
scripts/lib/wic/filemap.py +7 -0
@@ -563,6 +563,13 @@ def sparse_copy(src_fname, dst_fname, skip=0, seek=0, start = skip fmap._f_image.seek(start, os.SEEK_SET) + + written += start - skip - written + if length and written >= length: + dst_file.seek(seek + length, os.SEEK_SET) + dst_file.close() + return + dst_file.seek(seek + start - skip, os.SEEK_SET) chunk_size = 1024 * 1024
Signature · 1 line
2.1.4
PATCH 9/11 wic: add Disk._put_part_image method
In reply to Ed BartoshThis method copies partition image into the wic image. It will be used in 'wic cp' and 'wic rm' subcommands to copy changed partition back into wic image.
--- scripts/lib/wic/engine.py | 5 +++++ 1 file changed, 5 insertions(+)
scripts/lib/wic/engine.py +5 -0
@@ -300,6 +300,11 @@ class Disk: return self._partimages[pnum] + def _put_part_image(self, pnum): + """Put partition image into partitioned image.""" + sparse_copy(self._partimages[pnum], self.imagepath, + seek=self.partitions[pnum].start) + def dir(self, pnum, path): return exec_cmd("{} -i {} ::{}".format(self.mdir, self._get_part_image(pnum),
Signature · 1 line
2.1.4
PATCH 8/11 filemap: calculate dst size correctly
In reply to Ed BartoshFixed calculation of the dst file size using skip, seek and length parameters. Current code does it incorrectly which causes sparse_copy API to create unnecessary big output files.
--- scripts/lib/wic/filemap.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
scripts/lib/wic/filemap.py +5 -1
@@ -549,7 +549,11 @@ def sparse_copy(src_fname, dst_fname, skip=0, seek=0, dst_file = open(dst_fname, 'r+b') except IOError: dst_file = open(dst_fname, 'wb') - dst_file.truncate(os.path.getsize(src_fname)) + if length: + dst_size = length + seek + else: + dst_size = os.path.getsize(src_fname) + seek - skip + dst_file.truncate(dst_size) written = 0 for first, last in fmap.get_mapped_ranges(0, fmap.blocks_cnt):
Signature · 1 line
2.1.4
PATCH 10/11 wic: fully implement 'wic cp'
In reply to Ed BartoshAdded implementation of Disk.copy method and wic_cp function that copies files/directories to the vfat partition of the partitioned image.
[YOCTO #11283]
--- scripts/lib/wic/engine.py | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)
scripts/lib/wic/engine.py +10 -1
@@ -310,6 +310,14 @@ class Disk: self._get_part_image(pnum), path)) + def copy(self, src, pnum, path): + """Copy partition image into wic image.""" + cmd = "{} -i {} -snop {} ::{}".format(self.mcopy, + self._get_part_image(pnum), + src, path) + exec_cmd(cmd) + self._put_part_image(pnum) + def wic_ls(args, native_sysroot): """List contents of partitioned image or vfat partition.""" disk = Disk(args.path.image, native_sysroot)
@@ -329,7 +337,8 @@ def wic_cp(args, native_sysroot): Copy local file or directory to the vfat partition of partitioned image. """ - pass + disk = Disk(args.dest.image, native_sysroot) + disk.copy(args.src, args.dest.part, args.dest.path) def find_canned(scripts_path, file_name): """
Signature · 1 line
2.1.4
PATCH 11/11 selftest: add test_wic_cp test case
In reply to Ed BartoshAdded test case for "wic cp" functionality.
- copy file to vfat partition
- copy directory to vfat partition
--- meta/lib/oeqa/selftest/wic.py | 46 ++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 45 insertions(+), 1 deletion(-)
meta/lib/oeqa/selftest/wic.py +45 -1
@@ -28,7 +28,7 @@ import sys import unittest from glob import glob -from shutil import rmtree +from shutil import rmtree, copy from functools import wraps, lru_cache from tempfile import NamedTemporaryFile
@@ -845,3 +845,47 @@ part /etc --source rootfs --ondisk mmcblk0 --fstype=ext4 --exclude-path bin/ --r result = runCmd("wic ls %s:1/ -n %s" % (images[0], sysroot)) self.assertEqual(0, result.status) self.assertEqual(6, len(result.output.split('\n'))) + + def test_wic_cp(self): + """Test copy files and directories to the the wic image.""" + self.assertEqual(0, runCmd("wic create wictestdisk " + "--image-name=core-image-minimal " + "-D -o %s" % self.resultdir).status) + images = glob(self.resultdir + "wictestdisk-*.direct") + self.assertEqual(1, len(images)) + + sysroot = get_bb_var('RECIPE_SYSROOT_NATIVE', 'wic-tools') + + # list directory content of the first partition + result = runCmd("wic ls %s:1/ -n %s" % (images[0], sysroot)) + self.assertEqual(0, result.status) + self.assertEqual(6, len(result.output.split('\n'))) + + with NamedTemporaryFile("w", suffix=".wic-cp") as testfile: + testfile.write("test") + + # copy file to the partition + result = runCmd("wic cp %s %s:1/ -n %s" % (testfile.name, images[0], sysroot)) + self.assertEqual(0, result.status) + + # check if file is there + result = runCmd("wic ls %s:1/ -n %s" % (images[0], sysroot)) + self.assertEqual(0, result.status) + self.assertEqual(7, len(result.output.split('\n'))) + self.assertTrue(os.path.basename(testfile.name) in result.output) + + # prepare directory + testdir = os.path.join(self.resultdir, 'wic-test-cp-dir') + testsubdir = os.path.join(testdir, 'subdir') + os.makedirs(os.path.join(testsubdir)) + copy(testfile.name, testdir) + + # copy directory to the partition + result = runCmd("wic cp %s %s:1/ -n %s" % (testdir, images[0], sysroot)) + self.assertEqual(0, result.status) + + # check if directory is there + result = runCmd("wic ls %s:1/ -n %s" % (images[0], sysroot)) + self.assertEqual(0, result.status) + self.assertEqual(8, len(result.output.split('\n'))) + self.assertTrue(os.path.basename(testdir) in result.output)
Signature · 1 line
2.1.4
✗ patchtest: failure for This is an implementation of new wic subcommand 'wic cp'.
In reply to Ed Bartosh== Series Details ==
Series: This is an implementation of new wic subcommand 'wic cp'.
Revision: 1
URL : https://patchwork.openembedded.org/series/6907/
State : failure
== Summary ==
Thank you for submitting this patch series to OpenEmbedded Core. This is an automated response. Several tests have been executed on the proposed series by patchtest resulting in the following failures:
* Issue Series does not apply on top of target branch [test_series_merge_on_head]
Suggested fix Rebase your series on top of targeted branch
Targeted branch master (currently at 496a9dc179)
If you believe any of these test results are incorrect, please reply to the mailing list ([email protected]) raising your concerns. Otherwise we would appreciate you correcting the issues and submitting a new version of the patchset if applicable. Please ensure you add/increment the version number when sending the new version (i.e. [PATCH] -> [PATCH v2] -> [PATCH v3] -> ...).
---
Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest Test suite: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe