wic: implement 'include' command
cover wic: implement 'include' command
Hi,
Please review the implementation of 'include' command in wks parser.
This patchset depends on my previous patchset 'wic: kickstart fixes' which is in toaster-next, so I rebased this one on top of toaster-next.
The following changes since commit 429137c9d785be79816029b987d6102978e936e0:
classes/populate_sdk_ext: fix task dependency regression (2016-01-18 15:06:01 +0000)
are available in the git repository at:
git://git.yoctoproject.org/poky-contrib ed/wic/kickstart-include-8848
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/wic/kickstart-include-8848
Ed Bartosh (8):
wic: move wks parsing code to KickStart._parse
wic: use unique partition number
wic: do not remove build dir in source plugins
wic: ksparser: add support for include
wic: refactor get_boot_config
wic: implement search of includes
wic: move parts of canned .wks into common.wks.inc
wic: add help for 'include' command
scripts/lib/wic/canned-wks/common.wks.inc | 3 +++ .../canned-wks/directdisk-bootloader-config.wks | 4 +--- scripts/lib/wic/canned-wks/directdisk.wks | 4 +--- scripts/lib/wic/canned-wks/qemux86-directdisk.wks | 4 +--- scripts/lib/wic/help.py | 14 ++++++++++++ scripts/lib/wic/imager/direct.py | 3 +++ scripts/lib/wic/ksparser.py | 26 ++++++++++++++++++++-- scripts/lib/wic/plugins/source/bootimg-efi.py | 2 -- .../lib/wic/plugins/source/bootimg-partition.py | 3 --- scripts/lib/wic/plugins/source/bootimg-pcbios.py | 2 -- scripts/lib/wic/utils/misc.py | 16 ++++++------- 11 files changed, 54 insertions(+), 27 deletions(-) create mode 100644 scripts/lib/wic/canned-wks/common.wks.inc
--
Regards,
Ed
PATCH 1/8 wic: move wks parsing code to KickStart._parse
This is a preparation for implementation of include statement. Parser will be called recursively to parse included .wks files, so it should be available as a method.
--- scripts/lib/wic/ksparser.py | 3 +++ 1 file changed, 3 insertions(+)
scripts/lib/wic/ksparser.py +3 -0
@@ -115,6 +115,9 @@ class KickStart(object): bootloader.add_argument('--timeout', type=int) bootloader.add_argument('--source') + self._parse(parser, confpath) + + def _parse(self, parser, confpath): with open(confpath) as conf: lineno = 0 for line in conf:
Signature · 1 line
2.1.4
PATCH 2/8 wic: use unique partition number
In reply to Ed BartoshThis is a preparation for 'include' support.
Used unique counter instead of line number for partitions in .ks file. Line numbers can be equal for different .ks files, which can cause problems if one .ks file is included into another.
--- scripts/lib/wic/ksparser.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
scripts/lib/wic/ksparser.py +3 -1
@@ -84,6 +84,7 @@ class KickStart(object): self.partitions = [] self.bootloader = None self.lineno = 0 + self.partnum = 0 parser = KickStartParser() subparsers = parser.add_subparsers()
@@ -130,7 +131,8 @@ class KickStart(object): raise KickStartError('%s:%d: %s' % \ (confpath, lineno, err)) if line.startswith('part'): - self.partitions.append(Partition(parsed, lineno)) + self.partnum += 1 + self.partitions.append(Partition(parsed, self.partnum)) else: if not self.bootloader: self.bootloader = parsed
Signature · 1 line
2.1.4
PATCH 3/8 wic: do not remove build dir in source plugins
In reply to Ed BartoshInteresting bug was found during implementation of 'include' parser command.
Build directory was removed in do_configure_partition method of bootimg- source plugins. This can cause removal of previously prepared partition images if /boot partition is mentioned after other partitions in .ks file.
Moved work directory removal to direct.py before processing partitions.
--- scripts/lib/wic/imager/direct.py | 3 +++ scripts/lib/wic/plugins/source/bootimg-efi.py | 2 -- scripts/lib/wic/plugins/source/bootimg-partition.py | 3 --- scripts/lib/wic/plugins/source/bootimg-pcbios.py | 2 -- 4 files changed, 3 insertions(+), 7 deletions(-)
scripts/lib/wic/imager/direct.py +3 -0
@@ -229,6 +229,9 @@ class DirectImageCreator(BaseImageCreator): fstab_path = self._write_fstab(self.rootfs_dir.get("ROOTFS_DIR")) + shutil.rmtree(self.workdir) + os.mkdir(self.workdir) + for part in parts: # get rootfs size from bitbake variable if it's not set in .ks file if not part.size:
scripts/lib/wic/plugins/source/bootimg-efi.py +0 -2
@@ -142,8 +142,6 @@ class BootimgEFIPlugin(SourcePlugin): Called before do_prepare_partition(), creates loader-specific config """ hdddir = "%s/hdd/boot" % cr_workdir - rm_cmd = "rm -rf %s" % cr_workdir - exec_cmd(rm_cmd) install_cmd = "install -d %s/EFI/BOOT" % hdddir exec_cmd(install_cmd)
scripts/lib/wic/plugins/source/bootimg-partition.py +0 -3
@@ -71,9 +71,6 @@ class BootimgPartitionPlugin(SourcePlugin): - copies all files listed in IMAGE_BOOT_FILES variable """ hdddir = "%s/boot" % cr_workdir - rm_cmd = "rm -rf %s/boot" % cr_workdir - exec_cmd(rm_cmd) - install_cmd = "install -d %s" % hdddir exec_cmd(install_cmd)
scripts/lib/wic/plugins/source/bootimg-pcbios.py +0 -2
@@ -78,8 +78,6 @@ class BootimgPcbiosPlugin(SourcePlugin): Called before do_prepare_partition(), creates syslinux config """ hdddir = "%s/hdd/boot" % cr_workdir - rm_cmd = "rm -rf " + cr_workdir - exec_cmd(rm_cmd) install_cmd = "install -d %s" % hdddir exec_cmd(install_cmd)
Signature · 1 line
2.1.4
PATCH 4/8 wic: ksparser: add support for include
In reply to Ed BartoshExtended parser to support inclusion of .ks files:
recursively called self._parse to parse included .ks
--- scripts/lib/wic/ksparser.py | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
scripts/lib/wic/ksparser.py +6 -1
@@ -116,6 +116,9 @@ class KickStart(object): bootloader.add_argument('--timeout', type=int) bootloader.add_argument('--source') + include = subparsers.add_parser('include') + include.add_argument('path') + self._parse(parser, confpath) def _parse(self, parser, confpath):
@@ -133,7 +136,9 @@ class KickStart(object): if line.startswith('part'): self.partnum += 1 self.partitions.append(Partition(parsed, self.partnum)) - else: + elif line.startswith('include'): + self._parse(parser, parsed.path) + elif line.startswith('bootloader'): if not self.bootloader: self.bootloader = parsed else:
Signature · 1 line
2.1.4
PATCH 5/8 wic: refactor get_boot_config
In reply to Ed BartoshThis function is going to be used by ks parser to find include .wks files. get_boot_config name is a bit confusing as function is quite generic. It looks if file is present in the canned wks directories.
Renamed get_boot_config -> get_canned.
Renamed parameter file_boot -> file_name.
Updated description.
--- scripts/lib/wic/utils/misc.py | 16 +++++++--------- 1 file changed, 7 insertions(+), 9 deletions(-)
scripts/lib/wic/utils/misc.py +7 -9
@@ -58,25 +58,23 @@ def build_name(kscfg, release=None, prefix=None, suffix=None): return ret -def find_boot_config(scripts_path, boot_file): +def find_canned(scripts_path, file_name): """ - Find a config file with the given name in the canned files dir. + Find a file either by its path or by name in the canned files dir. - Return False if not found + Return None if not found """ - if os.path.exists(boot_file): - return boot_file + if os.path.exists(file_name): + return file_name layers_canned_wks_dir = wic.engine.build_canned_image_list(scripts_path) for canned_wks_dir in layers_canned_wks_dir: for root, dirs, files in os.walk(canned_wks_dir): for fname in files: - if fname == boot_file: + if fname == file_name: fullpath = os.path.join(canned_wks_dir, fname) return fullpath - return None - def get_custom_config(boot_file): """ Get the custom configuration to be used for the bootloader.
@@ -88,7 +86,7 @@ def get_custom_config(boot_file): for x in range(0, 3): scripts_path = os.path.dirname(scripts_path) - cfg_file = find_boot_config(scripts_path, boot_file) + cfg_file = find_canned(scripts_path, boot_file) if cfg_file: with open(cfg_file, "r") as f: config = f.read()
Signature · 1 line
2.1.4
PATCH 6/8 wic: implement search of includes
In reply to Ed BartoshUsed custom argument type to implement search of include .wks files in canned wks paths. Include files can be specified either by full path or by name.
[YOCTO #8848]
--- scripts/lib/wic/ksparser.py | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-)
scripts/lib/wic/ksparser.py +13 -1
@@ -30,6 +30,7 @@ import shlex from argparse import ArgumentParser, ArgumentError, ArgumentTypeError from wic.partition import Partition +from wic.utils.misc import find_canned class KickStartError(Exception): pass
@@ -78,6 +79,17 @@ def overheadtype(arg): return result +def cannedpathtype(arg): + """ + Custom type for ArgumentParser + Tries to find file in the list of canned wks paths + """ + scripts_path = os.path.abspath(os.path.dirname(__file__) + '../../..') + result = find_canned(scripts_path, arg) + if not result: + raise ArgumentTypeError("file not found: %s" % arg) + return result + class KickStart(object): def __init__(self, confpath):
@@ -117,7 +129,7 @@ class KickStart(object): bootloader.add_argument('--source') include = subparsers.add_parser('include') - include.add_argument('path') + include.add_argument('path', type=cannedpathtype) self._parse(parser, confpath)
Signature · 1 line
2.1.4
PATCH 7/8 wic: move parts of canned .wks into common.wks.inc
In reply to Ed BartoshIn order to give and example of 'include' feature of ks parser and for testing purposes common parts of 3 canned wks files were moved into common.wks.inc
--- scripts/lib/wic/canned-wks/common.wks.inc | 3 +++ scripts/lib/wic/canned-wks/directdisk-bootloader-config.wks | 4 +--- scripts/lib/wic/canned-wks/directdisk.wks | 4 +--- scripts/lib/wic/canned-wks/qemux86-directdisk.wks | 4 +--- 4 files changed, 6 insertions(+), 9 deletions(-) create mode 100644 scripts/lib/wic/canned-wks/common.wks.inc
scripts/lib/wic/canned-wks/common.wks.inc new file +3 -0
@@ -0,0 +1,3 @@ +# This file is included into 3 canned wks files from this directory +part /boot --source bootimg-pcbios --ondisk sda --label boot --active --align 1024 +part / --source rootfs --ondisk sda --fstype=ext4 --label platform --align 1024
scripts/lib/wic/canned-wks/directdisk-bootloader-config.wks +1 -3
@@ -2,9 +2,7 @@ # long-description: Creates a partitioned legacy BIOS disk image that the user # can directly dd to boot media. The bootloader configuration source is a user file. - -part /boot --source bootimg-pcbios --ondisk sda --label boot --active --align 1024 -part / --source rootfs --ondisk sda --fstype=ext4 --label platform --align 1024 +include common.wks.inc bootloader --configfile="directdisk-bootloader-config.cfg"
scripts/lib/wic/canned-wks/directdisk.wks +1 -3
@@ -2,9 +2,7 @@ # long-description: Creates a partitioned legacy BIOS disk image that the user # can directly dd to boot media. - -part /boot --source bootimg-pcbios --ondisk sda --label boot --active --align 1024 -part / --source rootfs --ondisk sda --fstype=ext4 --label platform --align 1024 +include common.wks.inc bootloader --timeout=0 --append="rootwait rootfstype=ext4 video=vesafb vga=0x318 console=tty0"
scripts/lib/wic/canned-wks/qemux86-directdisk.wks +1 -3
@@ -2,9 +2,7 @@ # long-description: Creates a partitioned legacy BIOS disk image that the user # can directly use to boot a qemu machine. - -part /boot --source bootimg-pcbios --ondisk sda --label boot --active --align 1024 -part / --source rootfs --ondisk sda --fstype=ext4 --label platform --align 1024 +include common.wks.inc bootloader --timeout=0 --append="vga=0 uvesafb.mode_option=640x480-32 root=/dev/vda2 rw mem=256M ip=192.168.7.2::192.168.7.1:255.255.255.0 oprofile.timer=1 rootfstype=ext4 "
Signature · 1 line
2.1.4
PATCH 8/8 wic: add help for 'include' command
In reply to Ed BartoshAdded description of 'include' parser command to the 'wic help kickstart' output.
--- scripts/lib/wic/help.py | 14 ++++++++++++++ 1 file changed, 14 insertions(+)
scripts/lib/wic/help.py +14 -0
@@ -760,4 +760,18 @@ DESCRIPTION implemented by the various --source plugins that implement bootloader functionality; the bootloader command essentially provides a means of modifying bootloader configuration. + + * include + + This command allows the user to include the content of .wks file + into original .wks file. + + Command uses the following syntax: + + include <file> + + The <file> is either path to the file or its name. If name is + specified wic will try to find file in the directories with canned + .wks files. + """
Signature · 1 line
2.1.4