image-bootfiles: new class
v2 cover image-bootfiles: new class
The image-bootfiles class is used to put all files listed in IMAGE_BOOT_FILES into the root filesystem.
IMAGE_BOOT_FILES is used by the bootimg-partition wic plugin to put the files into a boot partition.
Be able to list files as "boot files" in e.g. your .conf or image files instead of install those in every recipe is a good thing.
It is not always desired to have a separate boot partition for boot files. Sometimes it could be good to have them as a part of the root filesystem.
For example, if a double copy strategy is used for update the system, then you probably want to update both the boot files and root filesystem at the same time as there may be dependencies.
v2:
- Removed the documentation from the patch series (will be submitted later)
- Break out the parts in bootimg-partition that is used by image-bootfiles to a common library
- Make the destination directory in root filesystem configurable
Marcus Folkesson (2):
bootimg-partition: break out code to a common library. image-bootfiles.bbclass: new class, copy boot files to root filesystem
meta/classes/image-bootfiles.bbclass | 38 +++++++++++++ meta/lib/oe/bootfiles.py | 56 +++++++++++++++++++ .../wic/plugins/source/bootimg-partition.py | 39 +------------ 3 files changed, 97 insertions(+), 36 deletions(-) create mode 100644 meta/classes/image-bootfiles.bbclass create mode 100644 meta/lib/oe/bootfiles.py
Signature · 1 line
2.44.0
PATCH v2 1/2 bootimg-partition: break out code to a common library.
Break out the code that parse IMAGE_BOOT_FILES to a common library.
--- meta/lib/oe/bootfiles.py | 56 +++++++++++++++++++ .../wic/plugins/source/bootimg-partition.py | 39 +------------ 2 files changed, 59 insertions(+), 36 deletions(-) create mode 100644 meta/lib/oe/bootfiles.py
meta/lib/oe/bootfiles.py new file +56 -0
@@ -0,0 +1,56 @@ +# +# SPDX-License-Identifier: MIT +# +# Copyright (C) 2024 Marcus Folkesson +# Author: Marcus Folkesson <[email protected]> +# +# Utility functions handling boot files + +# Look into deploy_dir and search for boot_files. +# Returns a list with files to copy. +# +# Heavily inspired of bootimg-partition.py +# +def get_boot_files(deploy_dir, boot_files): + import re + import os + from glob import glob + + if boot_files is None: + return None + + # list of tuples (src_name, dst_name) + deploy_files = [] + for src_entry in re.findall(r'[\w;\-\./\*]+', boot_files): + if ';' in src_entry: + dst_entry = tuple(src_entry.split(';')) + if not dst_entry[0] or not dst_entry[1]: + raise ValueError('Malformed boot file entry: %s' % src_entry) + else: + dst_entry = (src_entry, src_entry) + + deploy_files.append(dst_entry) + + install_files = [] + for deploy_entry in deploy_files: + src, dst = deploy_entry + if '*' in src: + # by default install files under their basename + entry_name_fn = os.path.basename + if dst != src: + # unless a target name was given, then treat name + # as a directory and append a basename + entry_name_fn = lambda name: \ + os.path.join(dst, + os.path.basename(name)) + + srcs = glob(os.path.join(deploy_dir, src)) + + for entry in srcs: + src = os.path.relpath(entry, deploy_dir) + entry_dst_name = entry_name_fn(entry) + install_files.append((src, entry_dst_name)) + else: + install_files.append((src, dst)) + + return install_files
scripts/lib/wic/plugins/source/bootimg-partition.py +3 -36
@@ -18,6 +18,8 @@ import re from glob import glob +from oe.bootfiles import get_boot_files + from wic import WicError from wic.engine import get_custom_config from wic.pluginbase import SourcePlugin
@@ -66,42 +68,7 @@ class BootimgPartitionPlugin(SourcePlugin): logger.debug('Boot files: %s', boot_files) - # list of tuples (src_name, dst_name) - deploy_files = [] - for src_entry in re.findall(r'[\w;\-\./\*]+', boot_files): - if ';' in src_entry: - dst_entry = tuple(src_entry.split(';')) - if not dst_entry[0] or not dst_entry[1]: - raise WicError('Malformed boot file entry: %s' % src_entry) - else: - dst_entry = (src_entry, src_entry) - - logger.debug('Destination entry: %r', dst_entry) - deploy_files.append(dst_entry) - - cls.install_task = []; - for deploy_entry in deploy_files: - src, dst = deploy_entry - if '*' in src: - # by default install files under their basename - entry_name_fn = os.path.basename - if dst != src: - # unless a target name was given, then treat name - # as a directory and append a basename - entry_name_fn = lambda name: \ - os.path.join(dst, - os.path.basename(name)) - - srcs = glob(os.path.join(kernel_dir, src)) - - logger.debug('Globbed sources: %s', ', '.join(srcs)) - for entry in srcs: - src = os.path.relpath(entry, kernel_dir) - entry_dst_name = entry_name_fn(entry) - cls.install_task.append((src, entry_dst_name)) - else: - cls.install_task.append((src, dst)) - + cls.install_task = get_boot_files(kernel_dir, boot_files) if source_params.get('loader') != "u-boot": return
Signature · 1 line
2.44.0
Re: [PATCH v2 1/2] bootimg-partition: break out code to a common library.
Hi Marcus,
Quoted 16 lines · Marcus Folkesson
Break out the code that parse
IMAGE_BOOT_FILESto a common library.Signed-off-by Marcus Folkesson <[email protected]>--- meta/lib/oe/bootfiles.py | 56 +++++++++++++++++++ .../wic/plugins/source/bootimg-partition.py | 39 +------------ 2 files changed, 59 insertions(+), 36 deletions(-)create mode 100644
meta/lib/oe/bootfiles.pymeta/lib/oe/bootfiles.py new file +1
-0@@ -0,0 +1,56 @@ +#
meta/lib/oe/bootfiles.py line 11
Quoted 10 lines
meta/lib/oe/bootfiles.py new file +10
-0+# SPDX-License-Identifier: MIT +# +# Copyright (C) 2024 Marcus Folkesson +# Author: Marcus Folkesson <[email protected]> +# +# Utility functions handling boot files + +# Look into deploy_dir and search for boot_files. +# Returns a list with files to copy. +#
It returns a list of tuples with (original filepath relative to deploy_dir, desired filepath renaming) if I read the code properly, I think this is really important information.
Quoted 50 lines
meta/lib/oe/bootfiles.py new file +45
-0+# Heavily inspired of bootimg-partition.py +# +def get_boot_files(deploy_dir, boot_files): + import re + import os + from glob import glob + + if boot_files is None: + return None + + # list of tuples (src_name, dst_name) + deploy_files = [] + for src_entry in re.findall(r'[\w;\-\./\*]+', boot_files): + if ';' in src_entry: + dst_entry = tuple(src_entry.split(';')) + if not dst_entry[0] or not dst_entry[1]: + raise ValueError('Malformed boot file entry: %s' % src_entry) + else: + dst_entry = (src_entry, src_entry) + + deploy_files.append(dst_entry) + + install_files = [] + for deploy_entry in deploy_files: + src, dst = deploy_entry + if '*' in src: + # by default install files under their basename + entry_name_fn = os.path.basename + if dst != src: + # unless a target name was given, then treat name + # as a directory and append a basename + entry_name_fn = lambda name: \ + os.path.join(dst, + os.path.basename(name)) + + srcs = glob(os.path.join(deploy_dir, src)) + + for entry in srcs: + src = os.path.relpath(entry, deploy_dir) + entry_dst_name = entry_name_fn(entry) + install_files.append((src, entry_dst_name)) + else: + install_files.append((src, dst)) + + return install_filesscripts/lib/wic/plugins/source/bootimg-partition.py +0
-0
scripts/lib/wic/plugins/source/bootimg-partition.py import re · line 20
Quoted 3 lines
scripts/lib/wic/plugins/source/bootimg-partition.py +0
-0@@ -18,6 +18,8 @@ import re from glob import glob
You should be able to remove this import as it was only used in the code that is now moved to the lib.
Otherwise looks good to me, so
Thanks,
Quentin
Re: [PATCH v2 1/2] bootimg-partition: break out code to a common library.
Hi Quentin,
Quoted 1 line · Quentin Schulz
Hi Marcus,
[...]
Quoted 11 lines
Quoted 11 lines
meta/lib/oe/bootfiles.py new file +7
-0@@ -0,0 +1,56 @@ +# +# SPDX-License-Identifier: MIT +# +# Copyright (C) 2024 Marcus Folkesson +# Author: Marcus Folkesson <[email protected]> +# +# Utility functions handling boot files
Quoted 8 lines
meta/lib/oe/bootfiles.py line 11
Quoted 4 lines
meta/lib/oe/bootfiles.py new file +4
-0+ +# Look into deploy_dir and search for boot_files. +# Returns a list with files to copy. +#It returns a list of tuples with (original filepath relative to
deploy_dir, desired filepath renaming) if I read the code properly, I think this is really important information.
Agree. I will add that to the comment.
[...]
Quoted 4 lines
Quoted 1 line
from glob import glob
You should be able to remove this import as it was only used in the code that is now moved to the lib.
True, I will remove the import.
Quoted 7 lines
Otherwise looks good to me, so
Reviewed-by Quentin Schulz <[email protected]>Thanks,
Quentin
Best regards,
Marcus Folkesson
PATCH v2 2/2 image-bootfiles.bbclass: new class, copy boot files to root filesystem
In reply to Marcus Folkessonimage-bootfiles class copy files listed in IMAGE_BOOT_FILES to the IMAGE_BOOTFILES_DIR directory of the root filesystem.
This is useful when there is no explicit boot partition but all boot files should instead reside inside the root filesystem.
--- meta/classes/image-bootfiles.bbclass | 38 ++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+) create mode 100644 meta/classes/image-bootfiles.bbclass
meta/classes/image-bootfiles.bbclass new file +38 -0
@@ -0,0 +1,38 @@ +# +# SPDX-License-Identifier: MIT +# +# Copyright (C) 2024 Marcus Folkesson +# Author: Marcus Folkesson <[email protected]> +# +# Writes IMAGE_BOOT_FILES to the IMAGE_BOOTFILES_DIR directory +# +# +# Usage: add INHERIT += "image-bootfiles" to your image +# + +IMAGE_BOOTFILES_DIR ?= "/boot" + +def bootfiles_populate(d): + import shutil + from oe.bootfiles import get_boot_files + + boot_files = d.getVar("IMAGE_BOOT_FILES") + deploy_image_dir = d.getVar("DEPLOY_DIR_IMAGE") + boot_dir = d.getVar("IMAGE_ROOTFS") + d.getVar("IMAGE_BOOTFILES_DIR") + + install_files = get_boot_files(deploy_image_dir, boot_files) + if install_files is None: + return + + os.makedirs(boot_dir, exist_ok=True) + for entry in install_files: + src, dst = entry + image_src = os.path.join(deploy_image_dir, src) + image_dst = os.path.join(boot_dir, dst) + shutil.copyfile(image_src, image_dst) + +python bootfiles () { + bootfiles_populate(d), +} + +IMAGE_PREPROCESS_COMMAND += "bootfiles;"
Signature · 1 line
2.44.0
Re: [PATCH v2 2/2] image-bootfiles.bbclass: new class, copy boot files to root filesystem
Hi Marcus,
Quoted 19 lines · Marcus Folkesson
image-bootfiles class copy files listed in
IMAGE_BOOT_FILESto theIMAGE_BOOTFILES_DIRdirectory of the root filesystem.This is useful when there is no explicit boot partition but all boot files should instead reside inside the root filesystem.
Signed-off-by Marcus Folkesson <[email protected]>--- meta/classes/image-bootfiles.bbclass | 38 ++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+)create mode 100644 meta/classes/image-bootfiles.bbclass
meta/classes/image-bootfiles.bbclass new file +1
-0@@ -0,0 +1,38 @@ +#
meta/classes/image-bootfiles.bbclass line 11
Quoted 10 lines
meta/classes/image-bootfiles.bbclass new file +10
-0+# SPDX-License-Identifier: MIT +# +# Copyright (C) 2024 Marcus Folkesson +# Author: Marcus Folkesson <[email protected]> +# +# Writes IMAGE_BOOT_FILES to the IMAGE_BOOTFILES_DIR directory +# +# +# Usage: add INHERIT += "image-bootfiles" to your image +#
One is supposed to use inherit image-bootfiles inside recipes. INHERIT is for global inherit, from configuration files, e.g. INHERIT += "rm_work".
meta/classes/image-bootfiles.bbclass line 14
Quoted 3 lines
meta/classes/image-bootfiles.bbclass new file +3
-0+ +IMAGE_BOOTFILES_DIR ?= "/boot" +
I would really suggest to have the exact same prefix as for the IMAGE_BOOT_FILES variable since they are related. I would like to avoid the DEPLOYDIR, DEPLOY_DIR confusion for example :)
meta/classes/image-bootfiles.bbclass line 22
Quoted 8 lines
meta/classes/image-bootfiles.bbclass new file +8
-0+def bootfiles_populate(d): + import shutil + from oe.bootfiles import get_boot_files + + boot_files = d.getVar("IMAGE_BOOT_FILES") + deploy_image_dir = d.getVar("DEPLOY_DIR_IMAGE") + boot_dir = d.getVar("IMAGE_ROOTFS") + d.getVar("IMAGE_BOOTFILES_DIR") +
I would suggest to useos.path.join(d.getVar("IMAGE_ROOTFS"), d.getVar("IMAGE_BOOTFILES_DIR")) here to make it a tiny bit more robust to missing/redundant slashes (/) for example.
meta/classes/image-bootfiles.bbclass line 26
Quoted 4 lines
meta/classes/image-bootfiles.bbclass new file +4
-0+ install_files = get_boot_files(deploy_image_dir, boot_files) + if install_files is None: + return +
I'm wondering if we shouldn't print a warning or at the very least a note if boot_files is not empty but install_files is, this seems like there could be an issue no?
meta/classes/image-bootfiles.bbclass line 29
Quoted 3 lines
meta/classes/image-bootfiles.bbclass new file +3
-0+ os.makedirs(boot_dir, exist_ok=True) + for entry in install_files: + src, dst = entry
Those two lines could be merged into:
for src, dst in install_files:
I believe.
meta/classes/image-bootfiles.bbclass line 35
Quoted 6 lines
meta/classes/image-bootfiles.bbclass new file +6
-0+ image_src = os.path.join(deploy_image_dir, src) + image_dst = os.path.join(boot_dir, dst) + shutil.copyfile(image_src, image_dst) + +python bootfiles () { + bootfiles_populate(d),
I'm surprised by the comma at the end of the line, is this required somehow? What does this do?
I'm also not entirely sure we need to have this additional indirection? Is """
python bootfiles_populate() {
<code from def bootfiles_populate(d):>
}IMAGE_PREPROCESS_COMMAND += "bootfiles_populate;"
"""
not working somehow (I don't know if it should, so just asking)? tinyinitrd in meta/recipes-core/images/core-image-tiny-initramfs.bb seems to be doing it, so should/could be possible?
Cheers,
Quentin
Re: [PATCH v2 2/2] image-bootfiles.bbclass: new class, copy boot files to root filesystem
Hi Quentin,
Thank you for your in-depth review.
It is probably obvious that Python is not my mother tounge, so all feedback is appreciated :-)
Quoted 27 lines · Quentin Schulz
Hi Marcus,
Quoted 24 lines · Marcus Folkesson
image-bootfiles class copy files listed in
IMAGE_BOOT_FILESto theIMAGE_BOOTFILES_DIRdirectory of the root filesystem.This is useful when there is no explicit boot partition but all boot files should instead reside inside the root filesystem.
Signed-off-by Marcus Folkesson <[email protected]>--- meta/classes/image-bootfiles.bbclass | 38 ++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+)create mode 100644 meta/classes/image-bootfiles.bbclass
meta/classes/image-bootfiles.bbclass new file +6
-0@@ -0,0 +1,38 @@ +# +# SPDX-License-Identifier: MIT +# +# Copyright (C) 2024 Marcus Folkesson +# Author: Marcus Folkesson <[email protected]> +#
Quoted 8 lines
meta/classes/image-bootfiles.bbclass line 11
Quoted 5 lines
meta/classes/image-bootfiles.bbclass new file +5
-0+# Writes IMAGE_BOOT_FILES to the IMAGE_BOOTFILES_DIR directory +# +# +# Usage: add INHERIT += "image-bootfiles" to your image +#One is supposed to use inherit image-bootfiles inside recipes. INHERIT is for global inherit, from configuration files, e.g. INHERIT += "
rm_work".
Got it, thanks.
Quoted 8 lines
meta/classes/image-bootfiles.bbclass line 14
Quoted 4 lines
meta/classes/image-bootfiles.bbclass new file +3
-0+ +IMAGE_BOOTFILES_DIR ?= "/boot" +I would really suggest to have the exact same prefix as for the
IMAGE_BOOT_FILESvariable since they are related. I would like to avoid the DEPLOYDIR,DEPLOY_DIRconfusion for example :)
Good point.
Quoted 14 lines
meta/classes/image-bootfiles.bbclass line 22
Quoted 9 lines
meta/classes/image-bootfiles.bbclass new file +8
-0+def bootfiles_populate(d): + import shutil + from oe.bootfiles import get_boot_files + + boot_files = d.getVar("IMAGE_BOOT_FILES") + deploy_image_dir = d.getVar("DEPLOY_DIR_IMAGE") + boot_dir = d.getVar("IMAGE_ROOTFS") + d.getVar("IMAGE_BOOTFILES_DIR") +I would suggest to use
os.path.join(d.getVar("IMAGE_ROOTFS"), d.getVar("IMAGE_BOOTFILES_DIR"))here to make it a tiny bit more robust to missing/redundant slashes (/) for example.
That is better.
Quoted 9 lines
meta/classes/image-bootfiles.bbclass line 26
Quoted 5 lines
meta/classes/image-bootfiles.bbclass new file +4
-0+ install_files = get_boot_files(deploy_image_dir, boot_files) + if install_files is None: + return +I'm wondering if we shouldn't print a warning or at the very least a note if
boot_filesis not empty butinstall_filesis, this seems like there could be an issue no?
Sounds good, I will add a print.
Quoted 10 lines
meta/classes/image-bootfiles.bbclass line 29
Quoted 4 lines
meta/classes/image-bootfiles.bbclass new file +3
-0+ os.makedirs(boot_dir, exist_ok=True) + for entry in install_files: + src, dst = entryThose two lines could be merged into:
for src, dst in
install_files:I believe.
Yep, it worked.
Quoted 10 lines
meta/classes/image-bootfiles.bbclass line 35
Quoted 7 lines
meta/classes/image-bootfiles.bbclass new file +6
-0+ image_src = os.path.join(deploy_image_dir, src) + image_dst = os.path.join(boot_dir, dst) + shutil.copyfile(image_src, image_dst) + +python bootfiles () { + bootfiles_populate(d),I'm surprised by the comma at the end of the line, is this required somehow? What does this do?
No it is not. It is a remnant from my debugging. Good catch though.
Quoted 9 lines
I'm also not entirely sure we need to have this additional indirection? Is """
pythonbootfiles_populate(){
<code from defbootfiles_populate(d):>
}IMAGE_PREPROCESS_COMMAND+= "bootfiles_populate;"
"""
Quoted 3 lines
not working somehow (I don't know if it should, so just asking)? tinyinitrd in meta/recipes-core/images/core-image-tiny-initramfs.bb seems to be doing it, so should/could be possible?
I change it as suggested. bootfiles() did more things before I cleaned it up, so it is most of a remnant.
Quoted 3 lines
Cheers,
Quentin
Best regards,
Marcus Folkesson