Skip to contentSkip to conversation

image-bootfiles: new class

PATCH Superseded Marcus Folkesson 2 participants 7 messages May 25, 2024 – May 28, 2024

v2 of 6 Superseded Latest is v6 →

lore
Share & tools
lore
#0 lore raw

v2 cover image-bootfiles: new class

The image-bootfiles class is used to put all files listed in IMAGE_BOOT_FILES into the root filesystem.

IMAGE_BOOT_FILES is used by the bootimg-partition wic plugin to put the files into a boot partition.
Be able to list files as "boot files" in e.g. your .conf or image files instead of install those in every recipe is a good thing.

It is not always desired to have a separate boot partition for boot files. Sometimes it could be good to have them as a part of the root filesystem.

For example, if a double copy strategy is used for update the system, then you probably want to update both the boot files and root filesystem at the same time as there may be dependencies.

v2:
- Removed the documentation from the patch series (will be submitted later)
- Break out the parts in bootimg-partition that is used by image-bootfiles to a common library
- Make the destination directory in root filesystem configurable

Marcus Folkesson (2):
bootimg-partition: break out code to a common library. image-bootfiles.bbclass: new class, copy boot files to root filesystem

 meta/classes/image-bootfiles.bbclass          | 38 +++++++++++++
 meta/lib/oe/bootfiles.py                      | 56 +++++++++++++++++++
 .../wic/plugins/source/bootimg-partition.py   | 39 +------------
 3 files changed, 97 insertions(+), 36 deletions(-)
 create mode 100644 meta/classes/image-bootfiles.bbclass
 create mode 100644 meta/lib/oe/bootfiles.py
Signature · 1 line
2.44.0
#1 lore raw

PATCH v2 1/2 bootimg-partition: break out code to a common library.

Break out the code that parse IMAGE_BOOT_FILES to a common library.

---
 meta/lib/oe/bootfiles.py                      | 56 +++++++++++++++++++
 .../wic/plugins/source/bootimg-partition.py   | 39 +------------
 2 files changed, 59 insertions(+), 36 deletions(-)
 create mode 100644 meta/lib/oe/bootfiles.py
meta/lib/oe/bootfiles.py new file +56 -0
new file mode 100644
index 0000000000..81e09f138e
@@ -0,0 +1,56 @@
+#
+# SPDX-License-Identifier: MIT
+#
+# Copyright (C) 2024 Marcus Folkesson
+# Author: Marcus Folkesson <[email protected]>
+#
+# Utility functions handling boot files
+
+# Look into deploy_dir and search for boot_files.
+# Returns a list with files to copy.
+#
+# Heavily inspired of bootimg-partition.py
+#
+def get_boot_files(deploy_dir, boot_files):
+    import re
+    import os
+    from glob import glob
+
+    if boot_files is None:
+        return None
+
+    # list of tuples (src_name, dst_name)
+    deploy_files = []
+    for src_entry in re.findall(r'[\w;\-\./\*]+', boot_files):
+        if ';' in src_entry:
+            dst_entry = tuple(src_entry.split(';'))
+            if not dst_entry[0] or not dst_entry[1]:
+                raise ValueError('Malformed boot file entry: %s' % src_entry)
+        else:
+            dst_entry = (src_entry, src_entry)
+
+        deploy_files.append(dst_entry)
+
+    install_files = []
+    for deploy_entry in deploy_files:
+        src, dst = deploy_entry
+        if '*' in src:
+            # by default install files under their basename
+            entry_name_fn = os.path.basename
+            if dst != src:
+                # unless a target name was given, then treat name
+                # as a directory and append a basename
+                entry_name_fn = lambda name: \
+                                os.path.join(dst,
+                                             os.path.basename(name))
+
+            srcs = glob(os.path.join(deploy_dir, src))
+
+            for entry in srcs:
+                src = os.path.relpath(entry, deploy_dir)
+                entry_dst_name = entry_name_fn(entry)
+                install_files.append((src, entry_dst_name))
+        else:
+            install_files.append((src, dst))
+
+    return install_files
scripts/lib/wic/plugins/source/bootimg-partition.py +3 -36
index 1071d1af3f..b22a448b65 100644
@@ -18,6 +18,8 @@ import re
 
 from glob import glob
 
+from oe.bootfiles import get_boot_files
+
 from wic import WicError
 from wic.engine import get_custom_config
 from wic.pluginbase import SourcePlugin
@@ -66,42 +68,7 @@ class BootimgPartitionPlugin(SourcePlugin):
 
         logger.debug('Boot files: %s', boot_files)
 
-        # list of tuples (src_name, dst_name)
-        deploy_files = []
-        for src_entry in re.findall(r'[\w;\-\./\*]+', boot_files):
-            if ';' in src_entry:
-                dst_entry = tuple(src_entry.split(';'))
-                if not dst_entry[0] or not dst_entry[1]:
-                    raise WicError('Malformed boot file entry: %s' % src_entry)
-            else:
-                dst_entry = (src_entry, src_entry)
-
-            logger.debug('Destination entry: %r', dst_entry)
-            deploy_files.append(dst_entry)
-
-        cls.install_task = [];
-        for deploy_entry in deploy_files:
-            src, dst = deploy_entry
-            if '*' in src:
-                # by default install files under their basename
-                entry_name_fn = os.path.basename
-                if dst != src:
-                    # unless a target name was given, then treat name
-                    # as a directory and append a basename
-                    entry_name_fn = lambda name: \
-                                    os.path.join(dst,
-                                                 os.path.basename(name))
-
-                srcs = glob(os.path.join(kernel_dir, src))
-
-                logger.debug('Globbed sources: %s', ', '.join(srcs))
-                for entry in srcs:
-                    src = os.path.relpath(entry, kernel_dir)
-                    entry_dst_name = entry_name_fn(entry)
-                    cls.install_task.append((src, entry_dst_name))
-            else:
-                cls.install_task.append((src, dst))
-
+        cls.install_task = get_boot_files(kernel_dir, boot_files)
         if source_params.get('loader') != "u-boot":
             return
 
Signature · 1 line
2.44.0
Reviewed-by on v2 1/2
#2 lore raw

Re: [PATCH v2 1/2] bootimg-partition: break out code to a common library.

Hi Marcus,

Quoted 16 lines · Marcus Folkesson

Break out the code that parse IMAGE_BOOT_FILES to a common library.

---
  meta/lib/oe/bootfiles.py                      | 56 +++++++++++++++++++
  .../wic/plugins/source/bootimg-partition.py   | 39 +------------
  2 files changed, 59 insertions(+), 36 deletions(-)

create mode 100644 meta/lib/oe/bootfiles.py

meta/lib/oe/bootfiles.py new file +1 -0
new file mode 100644
index 0000000000..81e09f138e
@@ -0,0 +1,56 @@
+#

meta/lib/oe/bootfiles.py line 11

Quoted 10 lines
meta/lib/oe/bootfiles.py new file +10 -0
+# SPDX-License-Identifier: MIT
+#
+# Copyright (C) 2024 Marcus Folkesson
+# Author: Marcus Folkesson <[email protected]>
+#
+# Utility functions handling boot files
+
+# Look into deploy_dir and search for boot_files.
+# Returns a list with files to copy.
+#

It returns a list of tuples with (original filepath relative to deploy_dir, desired filepath renaming) if I read the code properly, I think this is really important information.

Quoted 50 lines
meta/lib/oe/bootfiles.py new file +45 -0
+# Heavily inspired of bootimg-partition.py
+#
+def get_boot_files(deploy_dir, boot_files):
+    import re
+    import os
+    from glob import glob
+
+    if boot_files is None:
+        return None
+
+    # list of tuples (src_name, dst_name)
+    deploy_files = []
+    for src_entry in re.findall(r'[\w;\-\./\*]+', boot_files):
+        if ';' in src_entry:
+            dst_entry = tuple(src_entry.split(';'))
+            if not dst_entry[0] or not dst_entry[1]:
+                raise ValueError('Malformed boot file entry: %s' % src_entry)
+        else:
+            dst_entry = (src_entry, src_entry)
+
+        deploy_files.append(dst_entry)
+
+    install_files = []
+    for deploy_entry in deploy_files:
+        src, dst = deploy_entry
+        if '*' in src:
+            # by default install files under their basename
+            entry_name_fn = os.path.basename
+            if dst != src:
+                # unless a target name was given, then treat name
+                # as a directory and append a basename
+                entry_name_fn = lambda name: \
+                                os.path.join(dst,
+                                             os.path.basename(name))
+
+            srcs = glob(os.path.join(deploy_dir, src))
+
+            for entry in srcs:
+                src = os.path.relpath(entry, deploy_dir)
+                entry_dst_name = entry_name_fn(entry)
+                install_files.append((src, entry_dst_name))
+        else:
+            install_files.append((src, dst))
+
+    return install_files
scripts/lib/wic/plugins/source/bootimg-partition.py +0 -0
index 1071d1af3f..b22a448b65 100644

scripts/lib/wic/plugins/source/bootimg-partition.py import re · line 20

Quoted 3 lines
scripts/lib/wic/plugins/source/bootimg-partition.py +0 -0
@@ -18,6 +18,8 @@ import re
 
  from glob import glob
 

You should be able to remove this import as it was only used in the code that is now moved to the lib.

Otherwise looks good to me, so

Thanks,
Quentin

#3 lore raw

Re: [PATCH v2 1/2] bootimg-partition: break out code to a common library.

Hi Quentin,

Quoted 1 line · Quentin Schulz

Hi Marcus,

[...]

Quoted 11 lines
Quoted 11 lines
meta/lib/oe/bootfiles.py new file +7 -0
@@ -0,0 +1,56 @@
+#
+# SPDX-License-Identifier: MIT
+#
+# Copyright (C) 2024 Marcus Folkesson
+# Author: Marcus Folkesson <[email protected]>
+#
+# Utility functions handling boot files
Quoted 8 lines

meta/lib/oe/bootfiles.py line 11

Quoted 4 lines
meta/lib/oe/bootfiles.py new file +4 -0
+
+# Look into deploy_dir and search for boot_files.
+# Returns a list with files to copy.
+#

It returns a list of tuples with (original filepath relative to deploy_dir, desired filepath renaming) if I read the code properly, I think this is really important information.

Agree. I will add that to the comment.

[...]

Quoted 4 lines
Quoted 1 line

from glob import glob

You should be able to remove this import as it was only used in the code that is now moved to the lib.

True, I will remove the import.

Quoted 7 lines

Otherwise looks good to me, so

Thanks,
Quentin

Best regards,
Marcus Folkesson

#4 lore raw

PATCH v2 2/2 image-bootfiles.bbclass: new class, copy boot files to root filesystem

In reply to Marcus Folkesson

image-bootfiles class copy files listed in IMAGE_BOOT_FILES to the IMAGE_BOOTFILES_DIR directory of the root filesystem.

This is useful when there is no explicit boot partition but all boot files should instead reside inside the root filesystem.

---
 meta/classes/image-bootfiles.bbclass | 38 ++++++++++++++++++++++++++++
 1 file changed, 38 insertions(+)
 create mode 100644 meta/classes/image-bootfiles.bbclass
meta/classes/image-bootfiles.bbclass new file +38 -0
new file mode 100644
index 0000000000..29a38ac631
@@ -0,0 +1,38 @@
+#
+# SPDX-License-Identifier: MIT
+#
+# Copyright (C) 2024 Marcus Folkesson
+# Author: Marcus Folkesson <[email protected]>
+#
+# Writes IMAGE_BOOT_FILES to the IMAGE_BOOTFILES_DIR directory
+#
+#
+# Usage: add INHERIT += "image-bootfiles" to your image
+#
+
+IMAGE_BOOTFILES_DIR ?= "/boot"
+
+def bootfiles_populate(d):
+    import shutil
+    from oe.bootfiles import get_boot_files
+
+    boot_files = d.getVar("IMAGE_BOOT_FILES")
+    deploy_image_dir = d.getVar("DEPLOY_DIR_IMAGE")
+    boot_dir = d.getVar("IMAGE_ROOTFS") + d.getVar("IMAGE_BOOTFILES_DIR")
+
+    install_files = get_boot_files(deploy_image_dir, boot_files)
+    if install_files is None:
+        return
+
+    os.makedirs(boot_dir, exist_ok=True)
+    for entry in install_files:
+        src, dst = entry
+        image_src = os.path.join(deploy_image_dir, src)
+        image_dst = os.path.join(boot_dir, dst)
+        shutil.copyfile(image_src, image_dst)
+
+python bootfiles () {
+   bootfiles_populate(d),
+}
+
+IMAGE_PREPROCESS_COMMAND += "bootfiles;"
Signature · 1 line
2.44.0
#5 lore raw

Re: [PATCH v2 2/2] image-bootfiles.bbclass: new class, copy boot files to root filesystem

Hi Marcus,

Quoted 19 lines · Marcus Folkesson

image-bootfiles class copy files listed in IMAGE_BOOT_FILES to the IMAGE_BOOTFILES_DIR directory of the root filesystem.

This is useful when there is no explicit boot partition but all boot files should instead reside inside the root filesystem.

---
  meta/classes/image-bootfiles.bbclass | 38 ++++++++++++++++++++++++++++
  1 file changed, 38 insertions(+)

create mode 100644 meta/classes/image-bootfiles.bbclass

meta/classes/image-bootfiles.bbclass new file +1 -0
new file mode 100644
index 0000000000..29a38ac631
@@ -0,0 +1,38 @@
+#

meta/classes/image-bootfiles.bbclass line 11

Quoted 10 lines
meta/classes/image-bootfiles.bbclass new file +10 -0
+# SPDX-License-Identifier: MIT
+#
+# Copyright (C) 2024 Marcus Folkesson
+# Author: Marcus Folkesson <[email protected]>
+#
+# Writes IMAGE_BOOT_FILES to the IMAGE_BOOTFILES_DIR directory
+#
+#
+# Usage: add INHERIT += "image-bootfiles" to your image
+#

One is supposed to use inherit image-bootfiles inside recipes. INHERIT is for global inherit, from configuration files, e.g. INHERIT += "rm_work".

meta/classes/image-bootfiles.bbclass line 14

Quoted 3 lines
meta/classes/image-bootfiles.bbclass new file +3 -0
+
+IMAGE_BOOTFILES_DIR ?= "/boot"
+

I would really suggest to have the exact same prefix as for the IMAGE_BOOT_FILES variable since they are related. I would like to avoid the DEPLOYDIR, DEPLOY_DIR confusion for example :)

meta/classes/image-bootfiles.bbclass line 22

Quoted 8 lines
meta/classes/image-bootfiles.bbclass new file +8 -0
+def bootfiles_populate(d):
+    import shutil
+    from oe.bootfiles import get_boot_files
+
+    boot_files = d.getVar("IMAGE_BOOT_FILES")
+    deploy_image_dir = d.getVar("DEPLOY_DIR_IMAGE")
+    boot_dir = d.getVar("IMAGE_ROOTFS") + d.getVar("IMAGE_BOOTFILES_DIR")
+

I would suggest to use
os.path.join(d.getVar("IMAGE_ROOTFS"), d.getVar("IMAGE_BOOTFILES_DIR")) here to make it a tiny bit more robust to missing/redundant slashes (/) for example.

meta/classes/image-bootfiles.bbclass line 26

Quoted 4 lines
meta/classes/image-bootfiles.bbclass new file +4 -0
+    install_files = get_boot_files(deploy_image_dir, boot_files)
+    if install_files is None:
+        return
+

I'm wondering if we shouldn't print a warning or at the very least a note if boot_files is not empty but install_files is, this seems like there could be an issue no?

meta/classes/image-bootfiles.bbclass line 29

Quoted 3 lines
meta/classes/image-bootfiles.bbclass new file +3 -0
+    os.makedirs(boot_dir, exist_ok=True)
+    for entry in install_files:
+        src, dst = entry

Those two lines could be merged into:

for src, dst in install_files:

I believe.

meta/classes/image-bootfiles.bbclass line 35

Quoted 6 lines
meta/classes/image-bootfiles.bbclass new file +6 -0
+        image_src = os.path.join(deploy_image_dir, src)
+        image_dst = os.path.join(boot_dir, dst)
+        shutil.copyfile(image_src, image_dst)
+
+python bootfiles () {
+   bootfiles_populate(d),

I'm surprised by the comma at the end of the line, is this required somehow? What does this do?

I'm also not entirely sure we need to have this additional indirection? Is """
python bootfiles_populate() {
<code from def bootfiles_populate(d):>
}
IMAGE_PREPROCESS_COMMAND += "bootfiles_populate;"
"""

not working somehow (I don't know if it should, so just asking)? tinyinitrd in meta/recipes-core/images/core-image-tiny-initramfs.bb seems to be doing it, so should/could be possible?

Cheers,
Quentin

#6 lore raw

Re: [PATCH v2 2/2] image-bootfiles.bbclass: new class, copy boot files to root filesystem

Hi Quentin,

Thank you for your in-depth review.
It is probably obvious that Python is not my mother tounge, so all feedback is appreciated :-)

Quoted 27 lines · Quentin Schulz

Hi Marcus,

Quoted 24 lines · Marcus Folkesson

image-bootfiles class copy files listed in IMAGE_BOOT_FILES to the IMAGE_BOOTFILES_DIR directory of the root filesystem.

This is useful when there is no explicit boot partition but all boot files should instead reside inside the root filesystem.

---
  meta/classes/image-bootfiles.bbclass | 38 ++++++++++++++++++++++++++++
  1 file changed, 38 insertions(+)

create mode 100644 meta/classes/image-bootfiles.bbclass

meta/classes/image-bootfiles.bbclass new file +6 -0
new file mode 100644
index 0000000000..29a38ac631
@@ -0,0 +1,38 @@
+#
+# SPDX-License-Identifier: MIT
+#
+# Copyright (C) 2024 Marcus Folkesson
+# Author: Marcus Folkesson <[email protected]>
+#
Quoted 8 lines

meta/classes/image-bootfiles.bbclass line 11

Quoted 5 lines
meta/classes/image-bootfiles.bbclass new file +5 -0
+# Writes IMAGE_BOOT_FILES to the IMAGE_BOOTFILES_DIR directory
+#
+#
+# Usage: add INHERIT += "image-bootfiles" to your image
+#

One is supposed to use inherit image-bootfiles inside recipes. INHERIT is for global inherit, from configuration files, e.g. INHERIT += "rm_work".

Got it, thanks.

Quoted 8 lines

meta/classes/image-bootfiles.bbclass line 14

Quoted 4 lines
meta/classes/image-bootfiles.bbclass new file +3 -0
+
+IMAGE_BOOTFILES_DIR ?= "/boot"
+

I would really suggest to have the exact same prefix as for the IMAGE_BOOT_FILES variable since they are related. I would like to avoid the DEPLOYDIR, DEPLOY_DIR confusion for example :)

Good point.

Quoted 14 lines

meta/classes/image-bootfiles.bbclass line 22

Quoted 9 lines
meta/classes/image-bootfiles.bbclass new file +8 -0
+def bootfiles_populate(d):
+    import shutil
+    from oe.bootfiles import get_boot_files
+
+    boot_files = d.getVar("IMAGE_BOOT_FILES")
+    deploy_image_dir = d.getVar("DEPLOY_DIR_IMAGE")
+    boot_dir = d.getVar("IMAGE_ROOTFS") + d.getVar("IMAGE_BOOTFILES_DIR")
+

I would suggest to use
os.path.join(d.getVar("IMAGE_ROOTFS"), d.getVar("IMAGE_BOOTFILES_DIR")) here to make it a tiny bit more robust to missing/redundant slashes (/) for example.

That is better.

Quoted 9 lines

meta/classes/image-bootfiles.bbclass line 26

Quoted 5 lines
meta/classes/image-bootfiles.bbclass new file +4 -0
+    install_files = get_boot_files(deploy_image_dir, boot_files)
+    if install_files is None:
+        return
+

I'm wondering if we shouldn't print a warning or at the very least a note if boot_files is not empty but install_files is, this seems like there could be an issue no?

Sounds good, I will add a print.

Quoted 10 lines

meta/classes/image-bootfiles.bbclass line 29

Quoted 4 lines
meta/classes/image-bootfiles.bbclass new file +3 -0
+    os.makedirs(boot_dir, exist_ok=True)
+    for entry in install_files:
+        src, dst = entry

Those two lines could be merged into:

for src, dst in install_files:

I believe.

Yep, it worked.

Quoted 10 lines

meta/classes/image-bootfiles.bbclass line 35

Quoted 7 lines
meta/classes/image-bootfiles.bbclass new file +6 -0
+        image_src = os.path.join(deploy_image_dir, src)
+        image_dst = os.path.join(boot_dir, dst)
+        shutil.copyfile(image_src, image_dst)
+
+python bootfiles () {
+   bootfiles_populate(d),

I'm surprised by the comma at the end of the line, is this required somehow? What does this do?

No it is not. It is a remnant from my debugging. Good catch though.

Quoted 9 lines

I'm also not entirely sure we need to have this additional indirection? Is """
python bootfiles_populate() {
<code from def bootfiles_populate(d):>
}
IMAGE_PREPROCESS_COMMAND += "bootfiles_populate;"
"""

Quoted 3 lines

not working somehow (I don't know if it should, so just asking)? tinyinitrd in meta/recipes-core/images/core-image-tiny-initramfs.bb seems to be doing it, so should/could be possible?

I change it as suggested. bootfiles() did more things before I cleaned it up, so it is most of a remnant.

Quoted 3 lines

Cheers,
Quentin

Best regards,
Marcus Folkesson