Skip to contentSkip to conversation

boot: add imagemap on-demand image loading from storage

PATCH Superseded Daniel Golle 3 participants 23 messages Aug 23 – Oct 1

Subsystems (2)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover boot: add imagemap on-demand image loading from storage

This series adds "imagemap", a small layer for reading FIT images directly from storage on demand instead of copying the whole image into RAM first. It keeps a translation table of the byte ranges already loaded, so a header probe, signature verification and the final load can reuse a single read, and it hooks transparently into fit_image_load() so all existing verification and decompression paths run unchanged for both RAM- and storage-backed images.

An imagemap device is created over a partition of a block device. That covers more than raw disks: an MTD partition is reached through mtdblock and a UBI volume through ubiblock, each exposed as a named block-device partition, so imagemap carries no storage-specific code of its own. NAND bad-block handling and wear-levelling stay with UBI, as they do elsewhere in U-Boot; on NOR a partition is read linearly with end-to-end integrity guaranteed by the FIT hash or signature.

To make those two paths reachable and testable, the first patches fill in the missing block devices: a non-NAND MTD master now gets an mtd_blk device automatically (patch 1), and attaching UBI now binds a ubi_blk device on any flash (patch 3, which makes the SPI-NAND-only eager binding redundant, dropped in patch 4).

The read path is built on the SPL struct spl_load_info abstraction and a new spl_load_region() helper. imagemap is full-U-Boot only; the SPL FIT loader is unaffected.

Testing (sandbox): "ut imagemap" (13 tests) covers the translation-table core plus real end-to-end reads of a NOR partition through mtd_blk and a UBI volume through ubiblock; a CONFIG_IMAGEMAP=n build is unaffected.

On hardware it has been tested end-to-end on a BananaPi R3 (mt7986), booting a dm-verity-protected rootfs from every storage path the board
exposes: eMMC and SD (block partition), SPI-NAND (UBI volume via ubiblock) and SPI-NOR (raw MTD partition via mtdblock). The first in-tree consumer, a bootmeth for OpenWrt-style FIT firmware, will follow as a separate series.

This series began as part of a combined RFC that also carried the OpenWrt boot method and its bootdevs [1]. Following the review there it has been reworked substantially:

- Build on U-Boot's existing SPL external-data loading path rather than a new abstraction: imagemap now uses struct spl_load_info and a new spl_load_region() helper (suggested by Marek Vasut).

- Do not extend the bootm command with a storage source; on-demand loading hooks into fit_image_load() transparently, and the in-tree consumer is a bootstd boot method reached through bootflow/bootdev/bootmeth rather than new bootm syntax (per Tom Rini).

- Move the storage-backed path out of the long fit_image_load() into its own helper, gated with if()/tools_build() instead of an inline #if (per Simon Glass).

- Collapse the three storage-specific backends (block, MTD and UBI) into a single block backend: NOR is read through mtd_blk and SPI-NAND through UBI/ubiblock, so imagemap holds no storage-specific code.

- Send the on-demand loading core on its own; the OpenWrt boot method, the MTD/UBI bootdevs and board enablement follow as separate series once this lands.

In order to test imagemap end-to-end with the intended bootmeth user I'm still relying on the downstream install-uuid mechanism of [2], which [3] could supersede. Both have stalled right now.

[1] https://lore.kernel.org/u-boot/[email protected]/
[2] https://github.com/open-source-firmware/flat-image-tree/pull/40
[3] https://github.com/open-source-firmware/flat-image-tree/pull/44

Based on u-boot/next.
Daniel Golle (9):
mtd: bind an mtd_blk device for non-NAND MTD masters
mtd: nand: sandbox: forbid sub-page writes
cmd: ubi: create a ubi_blk device when attaching UBI
mtd: nand: spi: drop the eager ubi_blk binding
boot: add imagemap on-demand loading from storage
boot: fit: support on-demand loading in fit_image_load()
configs: sandbox: enable imagemap and its storage backends
test: boot: add imagemap unit tests
doc: imagemap: document on-demand loading framework

 MAINTAINERS                      |   9 +
 arch/sandbox/dts/test.dts        |  29 ++
 boot/Kconfig                     |  16 +
 boot/Makefile                    |   2 +
 boot/bootm.c                     |  70 +++-
 boot/image-fit.c                 | 109 +++++-
 boot/imagemap.c                  | 451 +++++++++++++++++++++
 cmd/ubi.c                        |  27 ++
 configs/sandbox_defconfig        |   6 +
 doc/develop/imagemap.rst         | 100 +++++
 doc/develop/index.rst            |   1 +
 drivers/mtd/mtdcore.c            |  24 ++
 drivers/mtd/nand/raw/sand_nand.c |   2 +
 drivers/mtd/nand/spi/core.c      |   3 -
 include/bootm.h                  |   2 +
 include/dm/uclass-id.h           |   1 +
 include/image.h                  |   2 +
 include/imagemap.h               | 136 +++++++
 include/spl.h                    |  47 ++-
 test/boot/Makefile               |   2 +
 test/boot/imagemap.c             | 645 +++++++++++++++++++++++++++++++
 test/cmd_ut.c                    |   2 +
 22 files changed, 1671 insertions(+), 15 deletions(-)
 create mode 100644 boot/imagemap.c
 create mode 100644 doc/develop/imagemap.rst
 create mode 100644 include/imagemap.h
 create mode 100644 test/boot/imagemap.c

base-commit: 8841b7b292c74864acf4ed0bd8ce05839252c1f2

Signature · 1 line
2.55.0
#1 lore raw

PATCH 1/9 mtd: bind an mtd_blk device for non-NAND MTD masters

Only the SPI-NAND bind path currently creates an mtd_blk block device, so a bare NOR (parallel NOR, SPI-NOR, ...) has no block-device access at all. Bind an mtd_blk device for every non-NAND MTD master when CONFIG_MTD_BLOCK is enabled, so its partitions become reachable through the block layer (and thus by bootstd, imagemap, ...).

NAND is excluded on purpose: it needs UBI, which brings its own block device. Partitions inherit the master's block device via the "mtd" partition type, so only masters are bound.

mtd_bind() keeps a pointer to the passed mtd_info pointer (SPI-NAND fills it after bind), so the generic caller provides a small heap slot that lives as long as the block device; an MTD master is never removed in practice.

---
 drivers/mtd/mtdcore.c | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)
drivers/mtd/mtdcore.c +24 -0
index 3bfa5aebbc6..e0d0b5ae2e9 100644
@@ -494,6 +494,30 @@ int add_mtd_device(struct mtd_info *mtd)
 #endif
 
 	mutex_unlock(&mtd_table_mutex);
+
+	if (IS_ENABLED(CONFIG_MTD_BLOCK) && mtd->dev &&
+	    !mtd_is_partition(mtd) && !mtd_type_is_nand(mtd)) {
+		/*
+		 * Expose a non-NAND MTD master (parallel NOR, SPI-NOR, ...) as
+		 * a mtd_blk block device, so its partitions are reachable
+		 * through the block layer (and thus by imagemap, bootstd, ...).
+		 * NAND is excluded: it needs UBI, which brings its own block
+		 * device. Partitions inherit the master's block device via the
+		 * "mtd" partition type, so only masters are bound.
+		 *
+		 * mtd_bind() keeps a pointer to the passed mtd_info pointer, so
+		 * it needs storage that lives as long as the block device; the
+		 * MTD master is never removed in practice, so a small heap slot
+		 * is fine.
+		 */
+		struct mtd_info **mtdp = kmalloc(sizeof(*mtdp), GFP_KERNEL);
+
+		if (mtdp) {
+			*mtdp = mtd;
+			mtd_bind(mtd->dev, mtdp);
+		}
+	}
+
 	/* We _know_ we aren't being removed, because
 	   our caller is still holding us here. So none
 	   of this try_ nonsense, and no bitching about it
Signature · 1 line
2.55.0
#3 lore raw

PATCH 2/9 mtd: nand: sandbox: forbid sub-page writes

In reply to Daniel Golle

The sandbox NAND model programs each page in a single operation and rejects a second write to an already-programmed page. UBI writes the EC and VID headers into the first page in two separate sub-page operations, which the model fails with -EIO, so UBI cannot attach to a sandbox NAND.

Advertise NAND_NO_SUBPAGE_WRITE so callers place each header in its own page, matching what the model can actually do. This lets the imagemap UBI-volume unit test attach UBI on the sandbox NAND.

---
 drivers/mtd/nand/raw/sand_nand.c | 2 ++
 1 file changed, 2 insertions(+)
drivers/mtd/nand/raw/sand_nand.c +2 -0
index 3678bb8a41b..5c48b43a882 100644
@@ -602,6 +602,8 @@ static int sand_nand_probe(struct udevice *dev)
 
 		nand = &chip->nand;
 		nand->options = not_xpl() ? 0 : NAND_SKIP_BBTSCAN;
+		/* Each page is programmed in a single operation only */
+		nand->options |= NAND_NO_SUBPAGE_WRITE;
 		nand->flash_node = np;
 		nand->dev_ready = sand_nand_dev_ready;
 		nand->cmdfunc = sand_nand_command;
Signature · 1 line
2.55.0
#5 lore raw

PATCH 3/9 cmd: ubi: create a ubi_blk device when attaching UBI

In reply to Daniel Golle

A UBI volume can be reached as a partition on the ubi_blk block device, but that device is only bound from spinand_bind(), so it is missing on every other flash type (raw NAND, SPI-NOR, ...).

Bind a ubi_blk device when a UBI partition is attached, unless one has already been created, parented to the MTD device UBI sits on. UBI volumes then become reachable as block-device partitions regardless of the underlying flash. The imagemap block path relies on this to read a UBI volume.

---
 cmd/ubi.c | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)
cmd/ubi.c +27 -0
index 2b206141a21..91ba21504f4 100644
@@ -20,7 +20,10 @@
 #include <mtd.h>
 #include <nand.h>
 #include <onenand_uboot.h>
+#include <dm/device.h>
 #include <dm/devres.h>
+#include <dm/root.h>
+#include <dm/uclass-internal.h>
 #include <linux/mtd/mtd.h>
 #include <linux/mtd/partitions.h>
 #include <linux/err.h>
@@ -670,6 +673,29 @@ int ubi_detach(void)
 	return 0;
 }
 
+#if CONFIG_IS_ENABLED(UBI_BLOCK)
+static void ubi_blk_bind_once(void)
+{
+	struct udevice *dev;
+
+	/*
+	 * A single ubi_blk device serves all volumes of the attached UBI
+	 * device, the volume being selected through the block descriptor's
+	 * hwpart. Bind one when UBI is attached -- unless a previous attach
+	 * already did -- parented to the MTD device UBI sits on.
+	 */
+	for (uclass_find_first_device(UCLASS_BLK, &dev); dev;
+	     uclass_find_next_device(&dev)) {
+		if (dev->driver == DM_DRIVER_GET(ubi_blk))
+			return;
+	}
+
+	ubi_bind(ubi && ubi->mtd && ubi->mtd->dev ? ubi->mtd->dev : dm_root());
+}
+#else
+static inline void ubi_blk_bind_once(void) { }
+#endif
+
 int ubi_part(const char *part_name, const char *vid_header_offset)
 {
 	struct mtd_info *mtd;
@@ -698,6 +724,7 @@ int ubi_part(const char *part_name, const char *vid_header_offset)
 	}
 
 	ubi = ubi_devices[0];
+	ubi_blk_bind_once();
 
 	return 0;
 }
Signature · 1 line
2.55.0
#7 lore raw

PATCH 4/9 mtd: nand: spi: drop the eager ubi_blk binding

In reply to Daniel Golle

spinand_bind() binds a ubi_blk device at bind time. Now that "cmd:
ubi: create a ubi_blk device when attaching UBI" binds one generically when UBI is attached -- with an identical result, only lazily and for any flash type -- the SPI-NAND-specific eager call is redundant.

Drop it to avoid the duplicate code path (a ubi_blk is unusable before UBI is attached anyway). The mtd_blk binding is kept: bare mtd_blk access to SPI-NAND remains available for users who want it.

---
 drivers/mtd/nand/spi/core.c | 3 ---
 1 file changed, 3 deletions(-)
drivers/mtd/nand/spi/core.c +0 -3
index 14af4264612..fb8ca0a787d 100644
@@ -1654,9 +1654,6 @@ static int spinand_bind(struct udevice *dev)
 			if (ret)
 				return ret;
 		}
-
-		if (CONFIG_IS_ENABLED(UBI_BLOCK))
-			return ubi_bind(dev);
 	}
 
 	return 0;
Signature · 1 line
2.55.0
#8 lore raw

PATCH 5/9 boot: add imagemap on-demand loading from storage

In reply to Daniel Golle

Introduce imagemap, a small layer that reads image data from a storage device on demand and keeps a translation table of the byte ranges already loaded into RAM, instead of copying the whole image up front.

An imagemap device (UCLASS_IMAGEMAP) is created over a partition of a block device with imagemap_create(); imagemap_map() then returns a RAM pointer for any byte range, reading it in on first access and reusing it on later accesses -- so a header probe, verification and the final load of the same range share a single read. Block devices cover more than raw disks: an MTD partition is reached through mtdblock and a UBI volume through ubiblock, each exposed as a named block-device partition, so imagemap needs no storage-specific code of its own.

The read path is built on the SPL struct spl_load_info abstraction and a new spl_load_region() helper (added to spl.h): the block reader reports the device block length and spl_load_region() performs the native sector alignment. imagemap_map_to() keeps a payload byte-exact at a caller's load address, reading the aligned middle straight to the destination and bouncing only the partial head/tail block.

Regions that are not placed at a caller-supplied address are allocated through the LMB allocator and released on cleanup; the translation table doubles as the registry of those allocations. imagemap is gated to the full-U-Boot phase, so SPL and TPL carry no new cost.

---
 MAINTAINERS            |   7 +
 boot/Kconfig           |  16 ++
 boot/Makefile          |   2 +
 boot/imagemap.c        | 451 +++++++++++++++++++++++++++++++++++++++++
 include/dm/uclass-id.h |   1 +
 include/imagemap.h     | 136 +++++++++++++
 include/spl.h          |  47 ++++-
 7 files changed, 657 insertions(+), 3 deletions(-)
 create mode 100644 boot/imagemap.c
 create mode 100644 include/imagemap.h
MAINTAINERS +7 -0
index acaba95ed03..48ee01e033d 100644
@@ -1186,6 +1186,13 @@ S:	Maintained
 F:	drivers/timer/goldfish_timer.c
 F:	include/goldfish_timer.h
 
+IMAGEMAP
+M:	Daniel Golle <[email protected]>
+L:	[email protected]
+S:	Maintained
+F:	boot/imagemap.c
+F:	include/imagemap.h
+
 INTERCONNECT
 M:	Neil Armstrong <[email protected]>
 S:	Maintained
boot/Kconfig +16 -0
index c67dc0ba493..0d060aa25e4 100644
@@ -1205,6 +1205,22 @@ config SYS_BOOT_RAMDISK_HIGH
 
 endmenu		# Boot images
 
+config IMAGEMAP
+	bool "On-demand image loading from storage"
+	depends on DM && LMB && BLK && PARTITIONS
+	help
+	  Read image data from a storage device on demand instead of
+	  copying the whole image into RAM first. A translation table maps
+	  already-loaded byte ranges to their RAM addresses to avoid
+	  redundant reads, and scratch allocations are managed through the
+	  LMB allocator.
+
+	  Images are read from a partition on a block device, identified by
+	  number or name. MTD partitions (via mtdblock) and UBI volumes
+	  (via ubiblock) are reached as block-device partitions too. Used
+	  by bootm when a storage device is specified instead of a RAM
+	  address.
+
 config DISTRO_DEFAULTS
 	bool "(deprecated) Script-based booting of Linux distributions"
 	select CMDLINE
boot/Makefile +2 -0
index 7fb56e7ef37..b4200ebab3d 100644
@@ -73,6 +73,8 @@ obj-$(CONFIG_$(PHASE_)BOOTMETH_VBE_SIMPLE_OS) += vbe_simple_os.o
 
 obj-$(CONFIG_$(PHASE_)BOOTMETH_ANDROID) += bootmeth_android.o
 
+obj-$(CONFIG_$(PHASE_)IMAGEMAP) += imagemap.o
+
 obj-$(CONFIG_$(PHASE_)BOOTMETH_VBE_ABREC) += vbe_abrec.o vbe_common.o
 obj-$(CONFIG_$(PHASE_)BOOTMETH_VBE_ABREC_FW) += vbe_abrec_fw.o
 obj-$(CONFIG_$(PHASE_)BOOTMETH_VBE_ABREC_OS) += vbe_abrec_os.o
boot/imagemap.c new file +451 -0

451 lines not on this page · raw message

include/dm/uclass-id.h +1 -0
index fe0aae2720c..7de9312cde7 100644
@@ -75,6 +75,7 @@ enum uclass_id {
 	UCLASS_HASH,		/* Hash device */
 	UCLASS_HWSPINLOCK,	/* Hardware semaphores */
 	UCLASS_HOST,		/* Sandbox host device */
+	UCLASS_IMAGEMAP,	/* On-demand image loading from storage */
 	UCLASS_I2C,		/* I2C bus */
 	UCLASS_I2C_EEPROM,	/* I2C EEPROM device */
 	UCLASS_I2C_GENERIC,	/* Generic I2C device */
include/imagemap.h new file +136 -0

136 lines not on this page · raw message

include/spl.h +44 -3

74 lines not on this page · raw message

Signature · 1 line
2.55.0
#11 lore raw

PATCH 6/9 boot: fit: support on-demand loading in fit_image_load()

In reply to Daniel Golle

Integrate storage-backed image loading transparently into the existing FIT flow so that all verification, decompression, and copy-to-load- address code runs unchanged for both RAM-backed and storage-backed images.

Hook imagemap into fit_image_get_data(): when gd->imagemap is set and the sub-image uses external data, call imagemap_map() to bring the payload into RAM instead of returning the unmapped fit + offset pointer. Every existing caller, including fit_image_verify() called from fit_image_select(), transparently gets valid data without any changes to the authentication path.

For uncompressed sub-images with a known load address, pre-populate the translation table via imagemap_map_to() before verification runs. This loads data directly to the final RAM destination; when fit_image_get_data() is later called, the translation table hit returns the same pointer, achieving zero-copy. Compressed images or those without a load address fall through to the lazy imagemap_map() path.

Filesystem sub-images protected by dm-verity are returned early and stay on storage, as the kernel verifies them at block level.

---
 boot/bootm.c     |  70 +++++++++++++++++++++++++++---
 boot/image-fit.c | 109 ++++++++++++++++++++++++++++++++++++++++++++++-
 include/bootm.h  |   2 +
 include/image.h  |   2 +
 4 files changed, 174 insertions(+), 9 deletions(-)
boot/bootm.c +63 -7
index 3bce8586834..6422bdc16f6 100644
@@ -14,6 +14,7 @@
 #include <env.h>
 #include <errno.h>
 #include <fdt_support.h>
+#include <imagemap.h>
 #include <irq_func.h>
 #include <lmb.h>
 #include <log.h>
@@ -147,7 +148,22 @@ static int boot_get_kernel(const char *addr_fit, struct bootm_headers *images,
 
 	/* check image type, for FIT images get FIT kernel node */
 	*os_data = *os_len = 0;
-	buf = map_sysmem(img_addr, 0);
+	if (IS_ENABLED(CONFIG_IMAGEMAP) && images->imagemap) {
+		/*
+		 * Storage path: read enough bytes to detect the image
+		 * format. genimg_get_kernel_addr_fit() above still
+		 * parsed any #config / :subimage suffix so the FIT
+		 * selection variables are populated.
+		 */
+		buf = imagemap_map(images->imagemap, 0, 64);
+		if (IS_ERR(buf)) {
+			puts("Cannot read image header from storage\n");
+			return PTR_ERR(buf);
+		}
+		img_addr = map_to_sysmem(buf);
+	} else {
+		buf = map_sysmem(img_addr, 0);
+	}
 	switch (genimg_get_format(buf)) {
 #if CONFIG_IS_ENABLED(LEGACY_IMAGE_FORMAT)
 	case IMAGE_FORMAT_LEGACY:
@@ -193,6 +209,20 @@ static int boot_get_kernel(const char *addr_fit, struct bootm_headers *images,
 #endif
 #if CONFIG_IS_ENABLED(FIT)
 	case IMAGE_FORMAT_FIT:
+		if (IS_ENABLED(CONFIG_IMAGEMAP) && images->imagemap) {
+			/*
+			 * Extend the mapping to cover the full FIT
+			 * FDT structure so all metadata is accessible.
+			 */
+			size_t fdt_sz = fdt_totalsize(buf);
+
+			buf = imagemap_map(images->imagemap, 0, fdt_sz);
+			if (IS_ERR(buf)) {
+				puts("Cannot read FIT header from storage\n");
+				return PTR_ERR(buf);
+			}
+			img_addr = map_to_sysmem(buf);
+		}
 		os_noffset = fit_image_load(images, img_addr,
 				&fit_uname_kernel, &fit_uname_config,
 				IH_ARCH_DEFAULT, IH_TYPE_KERNEL,
@@ -1033,11 +1063,21 @@ int bootm_run_states(struct bootm_info *bmi, int states)
 	 * Work through the states and see how far we get. We stop on
 	 * any error.
 	 */
-	if (states & BOOTM_STATE_START)
+	if (states & BOOTM_STATE_START) {
 		ret = bootm_start();
+		/*
+		 * bootm_start() zeroes the global images struct. Restore
+		 * the loader pointer so the storage-backed path works.
+		 */
+		if (IS_ENABLED(CONFIG_IMAGEMAP) && bmi->imagemap)
+			images->imagemap = bmi->imagemap;
+	}
 
-	if (!ret && (states & BOOTM_STATE_PRE_LOAD))
-		ret = bootm_pre_load(bmi->addr_img);
+	if (!ret && (states & BOOTM_STATE_PRE_LOAD)) {
+		/* Pre-load verification is not applicable to storage boot */
+		if (!IS_ENABLED(CONFIG_IMAGEMAP) || !images->imagemap)
+			ret = bootm_pre_load(bmi->addr_img);
+	}
 
 	if (!ret && (states & BOOTM_STATE_FINDOS))
 		ret = bootm_find_os(bmi->cmd_name, bmi->addr_img);
@@ -1045,8 +1085,11 @@ int bootm_run_states(struct bootm_info *bmi, int states)
 	if (!ret && (states & BOOTM_STATE_FINDOTHER)) {
 		ulong img_addr;
 
-		img_addr = bmi->addr_img ? hextoul(bmi->addr_img, NULL)
-			: image_load_addr;
+		if (IS_ENABLED(CONFIG_IMAGEMAP) && images->imagemap)
+			img_addr = images->os.start;
+		else
+			img_addr = bmi->addr_img ? hextoul(bmi->addr_img, NULL)
+				: image_load_addr;
 		ret = bootm_find_other(img_addr, bmi->conf_ramdisk,
 				       bmi->conf_fdt);
 	}
@@ -1145,11 +1188,24 @@ int bootm_run_states(struct bootm_info *bmi, int states)
 	}
 
 	/* Now run the OS! We hope this doesn't return */
-	if (!ret && (states & BOOTM_STATE_OS_GO))
+	if (!ret && (states & BOOTM_STATE_OS_GO)) {
+		/* Release storage backend before jumping — no return expected */
+		if (IS_ENABLED(CONFIG_IMAGEMAP) && images->imagemap) {
+			imagemap_cleanup(images->imagemap);
+			images->imagemap = NULL;
+		}
+
 		ret = boot_selected_os(BOOTM_STATE_OS_GO, bmi, boot_fn);
+	}
 
 	/* Deal with any fallout */
 err:
+	/* Clean up imagemap on error (not reached on successful boot) */
+	if (IS_ENABLED(CONFIG_IMAGEMAP) && images->imagemap) {
+		imagemap_cleanup(images->imagemap);
+		images->imagemap = NULL;
+	}
+
 	if (iflag)
 		enable_interrupts();
 
boot/image-fit.c +107 -2

139 lines not on this page · raw message

include/bootm.h +2 -0
index f6958be751a..d1aac3d44df 100644
@@ -40,6 +40,7 @@ struct cmd_tbl;
  *	boot_get_fdt() for processing, or NULL for none
  * @boot_progress: true to show boot progress
  * @images: images information
+ * @imagemap: imagemap device for storage-backed boot (NULL for in-memory)
  * @cmd_name: command which invoked this operation, e.g. "bootm"
  * @argc: Number of arguments to the command (excluding the actual command).
  *	This is 0 if there are no arguments
@@ -51,6 +52,7 @@ struct bootm_info {
 	const char *conf_fdt;
 	bool boot_progress;
 	struct bootm_headers *images;
+	struct udevice *imagemap;
 	const char *cmd_name;
 	int argc;
 	char *const *argv;
include/image.h +2 -0
index 6edcb1995bf..f5baec9a516 100644
@@ -412,6 +412,8 @@ struct bootm_headers {
 
 	int		verify;		/* env_get("verify")[0] != 'n' */
 
+	struct udevice *imagemap;	/* on-demand storage loader, or NULL */
+
 #define BOOTM_STATE_START	0x00000001
 #define BOOTM_STATE_FINDOS	0x00000002
 #define BOOTM_STATE_FINDOTHER	0x00000004
Signature · 1 line
2.55.0
#13 lore raw

PATCH 7/9 configs: sandbox: enable imagemap and its storage backends

In reply to Daniel Golle

Enable CONFIG_IMAGEMAP and, so the end-to-end unit tests can exercise the real block paths, the storage that feeds them: CONFIG_MTD_BLOCK and CONFIG_SPI_FLASH_MTD (a bare non-NAND MTD reached via mtd_blk) and CONFIG_CMD_UBI / CONFIG_UBI_BLOCK (a UBI volume reached via ubi_blk).

Give the sandbox SPI-NOR two device-tree partitions for the mtd_blk test and add a clean sandbox NAND chip (nand@2, no injected errors) so UBI can format on it for the ubiblock test.

---
 arch/sandbox/dts/test.dts | 29 +++++++++++++++++++++++++++++
 configs/sandbox_defconfig |  6 ++++++
 2 files changed, 35 insertions(+)
arch/sandbox/dts/test.dts +29 -0
index bed2a4b65ab..b794a44a4e1 100644
@@ -1623,6 +1623,22 @@
 			compatible = "spansion,m25p16", "jedec,spi-nor";
 			spi-max-frequency = <40000000>;
 			sandbox,filename = "spi.bin";
+
+			partitions {
+				compatible = "fixed-partitions";
+				#address-cells = <1>;
+				#size-cells = <1>;
+
+				partition@0 {
+					label = "nor-fit";
+					reg = <0x0 0x100000>;
+				};
+
+				partition@100000 {
+					label = "nor-spare";
+					reg = <0x100000 0x100000>;
+				};
+			};
 		};
 		spi.bin@1 {
 			reg = <1>;
@@ -2232,6 +2248,19 @@
 			sandbox,err-count = <3>;
 			sandbox,err-step-size = <512>;
 		};
+
+		/* Clean chip (no injected errors) for UBI/ubiblock tests */
+		nand@2 {
+			reg = <2>;
+			nand-ecc-mode = "soft";
+			sandbox,id = [00 e3];
+			sandbox,erasesize = <(8 * 1024)>;
+			sandbox,oobsize = <16>;
+			sandbox,pagesize = <512>;
+			sandbox,pages = <0x2000>;
+			sandbox,err-count = <0>;
+			sandbox,err-step-size = <512>;
+		};
 	};
 
 	graph1 {
configs/sandbox_defconfig +6 -0
index f65e50590a3..05876cb36ba 100644
@@ -30,6 +30,7 @@ CONFIG_BOOTMETH_RAUC=y
 CONFIG_UPL=y
 CONFIG_LEGACY_IMAGE_FORMAT=y
 CONFIG_MEASURED_BOOT=y
+CONFIG_IMAGEMAP=y
 CONFIG_BOOTSTAGE=y
 CONFIG_BOOTSTAGE_REPORT=y
 CONFIG_BOOTSTAGE_FDT=y
@@ -399,3 +400,8 @@ CONFIG_UTHREAD=y
 CONFIG_UNIT_TEST=y
 CONFIG_UT_TIME=y
 CONFIG_UT_DM=y
+CONFIG_MTD_BLOCK=y
+CONFIG_SPI_FLASH_MTD=y
+CONFIG_CMD_UBI=y
+CONFIG_MTD_UBI=y
+CONFIG_UBI_BLOCK=y
Signature · 1 line
2.55.0
#15 lore raw

PATCH 8/9 test: boot: add imagemap unit tests

In reply to Daniel Golle

Add a test suite exercising imagemap:

- the translation-table core (map, lookup, map_to, cleanup, region growth, the block-aligned read path, and edge cases) using a mock RAM-backed reader, so most tests need no real storage;

- a NOR read through mtd_blk: write a known pattern into a device-tree partition of the sandbox SPI-NOR and read it back through imagemap, exercising imagemap -> spl_load_region -> blk_dread -> mtd_blk_read -> mtd_read for real;

- a UBI-volume read through ubiblock: attach UBI on the sandbox NAND, create a volume, write a pattern and read it back through imagemap on the ubi_blk device.

Registered under ut_boot_imagemap in cmd_ut.c.

---
 MAINTAINERS          |   1 +
 test/boot/Makefile   |   2 +
 test/boot/imagemap.c | 645 +++++++++++++++++++++++++++++++++++++++++++
 test/cmd_ut.c        |   2 +
 4 files changed, 650 insertions(+)
 create mode 100644 test/boot/imagemap.c
MAINTAINERS +1 -0
index 48ee01e033d..e8f0e94597f 100644
@@ -1192,6 +1192,7 @@ L:	[email protected]
 S:	Maintained
 F:	boot/imagemap.c
 F:	include/imagemap.h
+F:	test/boot/imagemap.c
 
 INTERCONNECT
 M:	Neil Armstrong <[email protected]>
test/boot/Makefile +2 -0
index 59a87028704..8dbecabde79 100644
@@ -27,3 +27,5 @@ endif
 obj-$(CONFIG_BOOTMETH_VBE) += vbe_fixup.o
 
 obj-$(CONFIG_UPL) += upl.o
+
+obj-$(CONFIG_IMAGEMAP) += imagemap.o
test/boot/imagemap.c new file +645 -0

645 lines not on this page · raw message

test/cmd_ut.c +2 -0
index 4328670d0d6..17610cdc2af 100644
@@ -73,6 +73,7 @@ SUITE_DECL(optee);
 SUITE_DECL(pci_mps);
 SUITE_DECL(seama);
 SUITE_DECL(setexpr);
+SUITE_DECL(imagemap);
 SUITE_DECL(upl);
 
 static struct suite suites[] = {
@@ -102,6 +103,7 @@ static struct suite suites[] = {
 	SUITE(pci_mps, "PCI Express Maximum Payload Size"),
 	SUITE(seama, "seama command parameters loading and decoding"),
 	SUITE(setexpr, "setexpr command"),
+	SUITE(imagemap, "imagemap on-demand storage loading"),
 	SUITE(upl, "Universal payload support"),
 };
 
Signature · 1 line
2.55.0
#17 lore raw

PATCH 9/9 doc: imagemap: document on-demand loading framework

In reply to Daniel Golle

Describe the imagemap framework architecture: the translation table that tracks mapped regions and its phase-determined caching, reading from a block-device partition (including MTD via mtdblock and UBI via ubiblock), the integration with bootm for on-demand FIT sub-image loading, and the SPL build.

---
 MAINTAINERS              |   1 +
 doc/develop/imagemap.rst | 100 +++++++++++++++++++++++++++++++++++++++
 doc/develop/index.rst    |   1 +
 3 files changed, 102 insertions(+)
 create mode 100644 doc/develop/imagemap.rst
MAINTAINERS +1 -0
index e8f0e94597f..f874443d87c 100644
@@ -1191,6 +1191,7 @@ M:	Daniel Golle <[email protected]>
 L:	[email protected]
 S:	Maintained
 F:	boot/imagemap.c
+F:	doc/develop/imagemap.rst
 F:	include/imagemap.h
 F:	test/boot/imagemap.c
 
doc/develop/imagemap.rst new file +100 -0

100 lines not on this page · raw message

doc/develop/index.rst +1 -0
index 0b757e8601e..3b4cc2e28a7 100644
@@ -31,6 +31,7 @@ Implementation
 
    directories
    bloblist
+   imagemap
    bootstd/index
    ci_testing
    commands
Signature · 1 line
2.55.0
#9 lore raw

Re: [PATCH 5/9] boot: add imagemap on-demand loading from storage

In reply to Daniel Golle
Quoted 27 lines · Daniel Golle

Introduce imagemap, a small layer that reads image data from a storage device on demand and keeps a translation table of the byte ranges already loaded into RAM, instead of copying the whole image up front.

An imagemap device (UCLASS_IMAGEMAP) is created over a partition of a block device with imagemap_create(); imagemap_map() then returns a RAM pointer for any byte range, reading it in on first access and reusing it on later accesses -- so a header probe, verification and the final load of the same range share a single read. Block devices cover more than raw disks: an MTD partition is reached through mtdblock and a UBI volume through ubiblock, each exposed as a named block-device partition, so imagemap needs no storage-specific code of its own.

The read path is built on the SPL struct spl_load_info abstraction and a new spl_load_region() helper (added to spl.h): the block reader reports the device block length and spl_load_region() performs the native sector alignment. imagemap_map_to() keeps a payload byte-exact at a caller's load address, reading the aligned middle straight to the destination and bouncing only the partial head/tail block.

Regions that are not placed at a caller-supplied address are allocated through the LMB allocator and released on cleanup; the translation table doubles as the registry of those allocations. imagemap is gated to the full-U-Boot phase, so SPL and TPL carry no new cost.

---

[snip]

include/spl.h typedef ulong (*spl_load_reader)(struct spl_load_info *load, ulong sector, · line 356

Quoted 12 lines
include/spl.h +4 -0
index 5078d7525ab..24df44d23e1 100644
@@ -350,10 +350,21 @@ typedef ulong (*spl_load_reader)(struct spl_load_info *load, ulong sector,
  * @phase: Image phase to load
  * @no_fdt_update: true to update the FDT with any loadables that are loaded
  */
+/*
+ * struct spl_load_info carries a device block length for the SPL block
+ * loaders (CONFIG_SPL_LOAD_BLOCK) and, in full U-Boot, for the imagemap
+ * on-demand loader (CONFIG_IMAGEMAP);

This part should be made clear in updates to the existing comment block.

Quoted 2 lines

imagemap is gated out of the xpl
+ * phases so SPL and TPL carry no new cost.

This doesn't need to be noted I think because..

include/spl.h typedef ulong (*spl_load_reader)(struct spl_load_info *load, ulong sector, · line 367

Quoted 11 lines

+ */

include/spl.h +6 -1
+#if IS_ENABLED(CONFIG_SPL_LOAD_BLOCK) || \
+	(IS_ENABLED(CONFIG_IMAGEMAP) && !defined(CONFIG_XPL_BUILD))
+#define SPL_LOAD_INFO_HAS_BL_LEN
+#endif
+
 struct spl_load_info {
 	spl_load_reader read;
 	void *priv;
-#if IS_ENABLED(CONFIG_SPL_LOAD_BLOCK)
+#ifdef SPL_LOAD_INFO_HAS_BL_LEN

Why not just:
#if IS_ENABLED(CONFIG_SPL_LOAD_BLOCK) || CONFIG_IS_ENABLED(IMAGEMAP)

? Then for when someone says "Ah, I want to do all of this right in SPL" there's less changes.

Signature · 1 line
Tom
#19 lore raw
In reply to Daniel Golle
Quoted 7 lines · Daniel Golle

This series adds "imagemap", a small layer for reading FIT images directly from storage on demand instead of copying the whole image into RAM first. It keeps a translation table of the byte ranges already loaded, so a header probe, signature verification and the final load can reuse a single read, and it hooks transparently into fit_image_load() so all existing verification and decompression paths run unchanged for both RAM- and storage-backed images.

As I was looking at the patches, I saw at least one checkpatch.pl alignment place to fix, so please cover those in v2.

Conceptually, I'm OK with this. I need to throw the series at a world build before/after and see what the fallout is on not-enabled platforms. I'm assuming that once this gets where I can throw Coverity at it, it will complain about a few things too.

Before posting v2, please throw this though CI too, I'm pretty sure I saw a documentation style problem that would blow up there. Thanks!

Signature · 1 line
Tom
#2 lore raw

Re: [PATCH 1/9] mtd: bind an mtd_blk device for non-NAND MTD masters

In reply to Daniel Golle

Hi Daniel,

Quoted 24 lines · Daniel Golle

mtd: bind an mtd_blk device for non-NAND MTD masters

Only the SPI-NAND bind path currently creates an mtd_blk block device, so a bare NOR (parallel NOR, SPI-NOR, ...) has no block-device access at all. Bind an mtd_blk device for every non-NAND MTD master when CONFIG_MTD_BLOCK is enabled, so its partitions become reachable through the block layer (and thus by bootstd, imagemap, ...).

NAND is excluded on purpose: it needs UBI, which brings its own block device. Partitions inherit the master's block device via the "mtd" partition type, so only masters are bound.

mtd_bind() keeps a pointer to the passed mtd_info pointer (SPI-NAND fills it after bind), so the generic caller provides a small heap slot that lives as long as the block device; an MTD master is never removed in practice.

drivers/mtd/mtdcore.c | 24 ++++++++++++++++++++++++

 1 file changed, 24 insertions(+)

drivers/mtd/mtdcore.c int add_mtd_device(struct mtd_info *mtd) · line 499

Quoted 6 lines
drivers/mtd/mtdcore.c +6 -0
@@ -494,6 +494,30 @@ int add_mtd_device(struct mtd_info *mtd)
+             struct mtd_info **mtdp = kmalloc(sizeof(*mtdp), GFP_KERNEL);
+
+             if (mtdp) {
+                     *mtdp = mtd;
+                     mtd_bind(mtd->dev, mtdp);
+             }

Two issues. The return value from mtd_bind() is dropped, so on failure mtdp is leaked (mtd_bind() only pr_err()s and returns; it doesn't free what was passed in). And if kmalloc() returns NULL the block device is silently not created, with no hint why partitions are unreachable. Please log a warning in both paths and free mtdp on the mtd_bind() failure path.

drivers/mtd/mtdcore.c int add_mtd_device(struct mtd_info *mtd) · line 498

Quoted 7 lines
drivers/mtd/mtdcore.c +5 -0
@@ -494,6 +494,30 @@ int add_mtd_device(struct mtd_info *mtd)
+              * mtd_bind() keeps a pointer to the passed mtd_info pointer, so
+              * it needs storage that lives as long as the block device; the
+              * MTD master is never removed in practice, so a small heap slot
+              * is fine.
+              */

The pointer-to-pointer indirection only exists because SPI-NAND fills its mtd in after bind. Every other caller (including this one) already knows the mtd, so the heap slot is pure overhead and, as the comment concedes, leaks if the master is ever removed. Since you're touching this path anyway, how about a second helper (say mtd_bind_master(struct udevice *, struct mtd_info *)) that stashes the mtd pointer itself, and leave the double-indirection variant to the one caller that needs it. What do you think?

Regards,
Simon

Reviewed-by on v1 2/9
#4 lore raw

Re: [PATCH 2/9] mtd: nand: sandbox: forbid sub-page writes

In reply to Daniel Golle

Hi Daniel,

Quoted 8 lines · Daniel Golle

mtd: nand: sandbox: forbid sub-page writes

The sandbox NAND model programs each page in a single operation and rejects a second write to an already-programmed page. UBI writes the EC and VID headers into the first page in two separate sub-page operations, which the model fails with -EIO, so UBI cannot attach to a sandbox NAND.

Quoted 8 lines

Advertise NAND_NO_SUBPAGE_WRITE so callers place each header in its own page, matching what the model can actually do. This lets the imagemap UBI-volume unit test attach UBI on the sandbox NAND.

drivers/mtd/nand/raw/sand_nand.c | 2 ++

 1 file changed, 2 insertions(+)

BTW, longer term it would be nicer for the sandbox model to actually support sub-page writes so we can exercise the code path UBI uses on real hardware, but that's a separate improvement.

Regards,
Simon

#6 lore raw

Re: [PATCH 3/9] cmd: ubi: create a ubi_blk device when attaching UBI

In reply to Daniel Golle

Hi Daniel,

Quoted 19 lines · Daniel Golle

cmd: ubi: create a ubi_blk device when attaching UBI

A UBI volume can be reached as a partition on the ubi_blk block device, but that device is only bound from spinand_bind(), so it is missing on every other flash type (raw NAND, SPI-NOR, ...).

Bind a ubi_blk device when a UBI partition is attached, unless one has already been created, parented to the MTD device UBI sits on. UBI volumes then become reachable as block-device partitions regardless of the underlying flash. The imagemap block path relies on this to read a UBI volume.

cmd/ubi.c | 27 +++++++++++++++++++++++++++

 1 file changed, 27 insertions(+)

cmd/ubi.c int ubi_detach(void) · line 674

Quoted 2 lines
cmd/ubi.c +2 -0
@@ -670,6 +673,29 @@ int ubi_detach(void)
+     ubi_bind(ubi && ubi->mtd && ubi->mtd->dev ? ubi->mtd->dev : dm_root());
+}

The return value from ubi_bind() is silently dropped, so a failed bind (e.g. -ENOMEM) leaves ubi_part() reporting success while there is no ubi_blk at all. Please propagate it, or at least log a warning.

cmd/ubi.c int ubi_detach(void) · line 683

Quoted 13 lines
cmd/ubi.c +11 -0
@@ -670,6 +673,29 @@ int ubi_detach(void)
+     /*
+      * A single ubi_blk device serves all volumes of the attached UBI
+      * device, the volume being selected through the block descriptor's
+      * hwpart. Bind one when UBI is attached -- unless a previous attach
+      * already did -- parented to the MTD device UBI sits on.
+      */
+     for (uclass_find_first_device(UCLASS_BLK, &dev); dev;
+          uclass_find_next_device(&dev)) {
+             if (dev->driver == DM_DRIVER_GET(ubi_blk))
+                     return;
+     }

ubi_detach() doesn't unbind the ubi_blk, so after 'ubi part A; ubi detach; ubi part B' the existing ubi_blk stays parented to A's MTD device even though UBI now sits on B. Reads still work because get_ubi_device() returns ubi_devices[0], but the DM tree is left with a stale parent (and A's MTD isn't safe to remove). Either unbind in ubi_detach(), or re-parent here when the target MTD has changed.

Also, walking every UCLASS_BLK to detect a single ubi_blk is awkward. Since ubi_blk is always parented to an MTD device, iterating the children of that parent e.g. with device_find_first_child_by_uclass() (or tracking the bound udevice in a static pointer) would be more direct and avoid poking dev->driver.

Regards,
Simon

#12 lore raw

Re: [PATCH 6/9] boot: fit: support on-demand loading in fit_image_load()

In reply to Daniel Golle

Hi Daniel,

Quoted 30 lines · Daniel Golle

boot: fit: support on-demand loading in fit_image_load()

Integrate storage-backed image loading transparently into the existing FIT flow so that all verification, decompression, and copy-to-load- address code runs unchanged for both RAM-backed and storage-backed images.

Hook imagemap into fit_image_get_data(): when gd->imagemap is set and the sub-image uses external data, call imagemap_map() to bring the payload into RAM instead of returning the unmapped fit + offset pointer. Every existing caller, including fit_image_verify() called from fit_image_select(), transparently gets valid data without any changes to the authentication path.

For uncompressed sub-images with a known load address, pre-populate the translation table via imagemap_map_to() before verification runs. This loads data directly to the final RAM destination; when fit_image_get_data() is later called, the translation table hit returns the same pointer, achieving zero-copy. Compressed images or those without a load address fall through to the lazy imagemap_map() [...]

boot/bootm.c | 70 +++++++++++++++++++++++++++++++----

 boot/image-fit.c | 109 ++++++++++++++++++++++++++++++++++++++++++++++++++++++-
 include/bootm.h  |   2 +
 include/image.h  |   2 +
 4 files changed, 174 insertions(+), 9 deletions(-)

boot/bootm.c static int boot_get_kernel(const char *addr_fit, struct bootm_headers *images, · line 148

Quoted 1 line
boot/bootm.c +1 -0
@@ -147,7 +148,22 @@ static int boot_get_kernel(const char *addr_fit, struct bootm_headers *images,
+             buf = imagemap_map(images->imagemap, 0, 64);

Where does 64 come from? A short comment or named constant would help
- genimg_get_format() has to distinguish a legacy header, a FIT FDT magic and possibly an Android header, each with different minimum sizes, so an unexplained 64 looks like a coincidence.

boot/bootm.c int bootm_run_states(struct bootm_info *bmi, int states) · line 1071

Quoted 11 lines
boot/bootm.c +8 -0
@@ -1033,11 +1063,21 @@ int bootm_run_states(struct bootm_info *bmi, int states)
+     if (states & BOOTM_STATE_START) {
              ret = bootm_start();
+             /*
+              * bootm_start() zeroes the global images struct. Restore
+              * the loader pointer so the storage-backed path works.
+              */
+             if (IS_ENABLED(CONFIG_IMAGEMAP) && bmi->imagemap)
+                     images->imagemap = bmi->imagemap;
+     }

Restoring a field immediately after bootm_start() zeroes it is fragile
- any future state that also calls bootm_start() (BOOTM_STATE_RESTART already runs bootm_restart(), similar in spirit) will silently drop the pointer again. Please move this into bootm_start() itself, or teach bootm_start() to preserve imagemap across the reset, so the invariant is expressed in one place.

boot/bootm.c int bootm_run_states(struct bootm_info *bmi, int states) · line 1193

Quoted 8 lines
boot/bootm.c +6 -0
@@ -1145,11 +1188,24 @@ int bootm_run_states(struct bootm_info *bmi, int states)
+err:
+     /* Clean up imagemap on error (not reached on successful boot) */
+     if (IS_ENABLED(CONFIG_IMAGEMAP) && images->imagemap) {
+             imagemap_cleanup(images->imagemap);
+             images->imagemap = NULL;
+     }

Several paths in bootm_run_states() return directly rather than goto err - 'if (images->no_os) return -ENOPKG;', 'if (ret) return ret;' after the OS-func lookup, and the 'ERROR: booting os …' return 1 all bypass this cleanup, so imagemap leaks on those exits. Please convert those to goto err, or drive the cleanup from a single place that runs on every exit.

boot/image-fit.c int fit_image_get_data(const void *fit, int noffset, const void **data, · line 1150

Quoted 12 lines
boot/image-fit.c +10 -0
@@ -1140,6 +1141,16 @@ int fit_image_get_data(const void *fit, int noffset, const void **data,
+#if !defined(USE_HOSTCC) && CONFIG_IS_ENABLED(IMAGEMAP)
+                     if (images.imagemap) {
+                             void *mapped;
+
+                             mapped = imagemap_lookup(images.imagemap,
+                                                         offset, len);
+                             if (mapped)
+                                     *data = mapped;
+                     }
+#endif

Reaching into the global images from a low-level helper couples fit_image_get_data() to bootm state - until now this function has been usable purely from (fit, noffset). Please pass the imagemap in explicitly (as a parameter or via a small wrapper), which also keeps 'ut fit' style tests free of hidden global dependencies.

boot/image-fit.c static const char *fit_get_image_type_property(int ph_type) · line 2156

Quoted 8 lines
boot/image-fit.c +6 -0
@@ -2140,6 +2151,84 @@ static const char *fit_get_image_type_property(int ph_type)
+     if (CONFIG_IS_ENABLED(FIT_VERITY)) {
+             u8 img_type;
+
+             if (!fit_image_get_type(fit, noffset, &img_type) &&
+                 img_type == IH_TYPE_FILESYSTEM &&
+                 fdt_subnode_offset(fit, noffset, "dm-verity") >= 0) {

Please use FIT_VERITY_NODENAME from image.h rather than a literal dm-verity, so there is one place to change the name.

boot/image-fit.c static const char *fit_get_image_type_property(int ph_type) · line 2159

Quoted 11 lines
boot/image-fit.c +9 -0
@@ -2140,6 +2151,84 @@ static const char *fit_get_image_type_property(int ph_type)
+     if (!fit_image_get_data_position(fit, noffset, &data_off)) {
+             external = true;
+     } else if (!fit_image_get_data_offset(fit, noffset, &data_off)) {
+             external = true;
+             data_off += ALIGN(fdt_totalsize(fit), 4);
+     }
+
+     if (!external || fit_image_get_data_size(fit, noffset, &data_sz))
+             return 0;

This re-implements the data-position / data-offset / totalsize-align dance that fit_image_get_data() already does a few hundred lines up, and uses a slightly different expression (ALIGN(fdt_totalsize(fit), 4) vs ((fdt_totalsize(fit) + 3) & ~3)). Please extract a shared helper - otherwise a future fix in one path (e.g. the INT_MAX bounds check that fit_image_get_data() has) will silently miss this one.

boot/image-fit.c static const char *fit_get_image_type_property(int ph_type) · line 2156

Quoted 8 lines
boot/image-fit.c +6 -0
@@ -2140,6 +2151,84 @@ static const char *fit_get_image_type_property(int ph_type)
+     if (img_comp == IH_COMP_NONE && load_op != FIT_LOAD_IGNORED &&
+         !fit_image_get_load(fit, noffset, &img_load)) {
+             void *dst = map_sysmem(img_load, data_sz);
+
+             mapped = imagemap_map_to(images->imagemap, data_off, data_sz,
+                                      dst);

map_sysmem() should really be paired with unmap_sysmem() - a no-op on most arches but not on sandbox, which is where the tests run.

boot/image-fit.c int fit_image_load(struct bootm_headers *images, ulong addr, · line 2455

Quoted 7 lines
boot/image-fit.c +3 -2
@@ -2349,8 +2453,9 @@ int fit_image_load(struct bootm_headers *images, ulong addr,
-             printf("   Loading %s from 0x%08lx to 0x%08lx\n",
-                    prop_name, data, load);
+             if (!CONFIG_IS_ENABLED(IMAGEMAP) || data != load)
+                     printf("   Loading %s from 0x%08lx to 0x%08lx\n",
+                            prop_name, data, load);

This is a build-time gate, not a runtime one - any board with CONFIG_IMAGEMAP=y silently loses the 'Loading … from … to …' message for RAM-backed FITs whenever data happens to equal load, a user-visible regression for boards that don't use imagemap at boot. Please key the suppression on 'images->imagemap && data == load' so the message only disappears on the genuine zero-copy path.

include/image.h struct bootm_headers { · line 412

Quoted 3 lines
include/image.h +1 -0
@@ -412,6 +412,8 @@ struct bootm_headers {
+     struct udevice *imagemap;       /* on-demand storage loader, or NULL */

It might be worth making this (and the equivalent field in struct bootm_info) conditional on CONFIG_IMAGEMAP, in line with the surrounding fields - otherwise every board pays the pointer with the feature off.

Regards,
Simon

#10 lore raw

Re: [PATCH 5/9] boot: add imagemap on-demand loading from storage

In reply to Daniel Golle

Hi Daniel,

Quoted 47 lines · Daniel Golle

boot: add imagemap on-demand loading from storage

Introduce imagemap, a small layer that reads image data from a storage device on demand and keeps a translation table of the byte ranges already loaded into RAM, instead of copying the whole image up front.

An imagemap device (UCLASS_IMAGEMAP) is created over a partition of a block device with imagemap_create(); imagemap_map() then returns a RAM pointer for any byte range, reading it in on first access and reusing it on later accesses -- so a header probe, verification and the final load of the same range share a single read. Block devices cover more than raw disks: an MTD partition is reached through mtdblock and a UBI volume through ubiblock, each exposed as a named block-device partition, so imagemap needs no storage-specific code of its own.

The read path is built on the SPL struct spl_load_info abstraction and a new spl_load_region() helper (added to spl.h): the block reader reports the device block length and spl_load_region() performs the native sector alignment. imagemap_map_to() keeps a payload byte-exact at a caller's load address, reading the aligned middle straight to the [...]

MAINTAINERS | 7 +

 boot/Kconfig           |  16 ++
 boot/Makefile          |   2 +
 boot/imagemap.c        | 451 +++++++++++++++++++++++++++++++++++++++++++++++++
 include/dm/uclass-id.h |   1 +
 include/imagemap.h     | 136 +++++++++++++++
 include/spl.h          |  47 +++++-
 7 files changed, 657 insertions(+), 3 deletions(-)
boot/imagemap.c +14 -0
@@ -0,0 +1,451 @@
+void *imagemap_map_to(struct udevice *dev, loff_t img_offset,
+                   ulong size, void *dst)
+{
+     struct imagemap_priv *priv = dev_get_uclass_priv(dev);
+     void *p;
+     int ret;
+
+     /* If already mapped to this exact destination, return it */
+     p = imagemap_lookup(dev, img_offset, size);
+     if (p && p == dst)
+             return p;
+
+     ret = imagemap_read_exact(&priv->info, img_offset, size, dst);
+     if (ret) {

boot/imagemap.c line 24

Quoted 10 lines
boot/imagemap.c +10 -0
+             log_err("imagemap: read failed at offset 0x%llx (size 0x%lx): %d\n",
+                     (unsigned long long)img_offset, size, ret);
+             return ERR_PTR(ret);
+     }
+
+     if (!imagemap_record(dev, img_offset, size, dst, false))
+             return ERR_PTR(-ENOMEM);
+
+     return dst;
+}

If an earlier imagemap_map() has already recorded a region at the same img_offset with lmb_reserved=true, this calls imagemap_record() with lmb_reserved=false, overwriting the entry in place - so it looks like the LMB reservation is silently orphaned and never freed by imagemap_cleanup(). Please free the old reservation before overwriting or refuse the overwrite.

boot/imagemap.c line 10

Quoted 12 lines
boot/imagemap.c +10 -0
@@ -0,0 +1,451 @@
+     if (bl_len == 1)
+             return spl_load_region(info, off, size, dst) < 0 ? -EIO : 0;
+
+     /* Partial head block: bounce and copy the wanted tail of the block */
+     if (overhead) {
+             ALLOC_CACHE_ALIGN_BUFFER(u8, blk, bl_len);
+             ulong chunk = min(left, bl_len - overhead);
+
+             if (info->read(info, cur - overhead, bl_len, blk) < bl_len)
+                     return -EIO;

ALLOC_CACHE_ALIGN_BUFFER() with a runtime bl_len expands to a stack VLA. For MMC that's 512 bytes, but a UBI/ubiblock leb_size or MTD erase-block-sized reader can push this into the KB range. Probably this is fine, but do you know what the upper limit will be?

Quoted 5 lines
boot/imagemap.c +3 -0
@@ -0,0 +1,451 @@
+     alist_for_each(r, &priv->regions) {
+             if (r->img_offset == base_off && r->size < read_size) {
+                     addr = map_to_sysmem(r->ram);

boot/imagemap.c line 13

Quoted 10 lines
boot/imagemap.c +10 -0
+
+                     /* Free old LMB reservation if we own it */
+                     if (r->lmb_reserved)
+                             lmb_free(addr,
+                                      ALIGN(r->size, ARCH_DMA_MINALIGN),
+                                      LMB_NONE);
+
+                     /* Try to re-reserve at the same address with new size */
+                     if (lmb_alloc_mem(LMB_MEM_ALLOC_ADDR, 0, &addr,
+                                       alloc_size, LMB_NONE)) {

Two concerns with this extend path:

1. If r->lmb_reserved is false (the region was recorded by imagemap_map_to() at a caller-supplied dst), this happily lmb_alloc_mem()s over that memory and then re-reads on top of it. Whatever the caller placed there is silently clobbered. Please skip the extend path in that case, or refuse to extend a non-LMB region.

2. Even in the LMB-owned case, when the in-place re-reservation succeeds the whole range gets re-read although the first r->size bytes are already valid - the whole point of the translation table is to avoid redundant reads. Only the newly-covered tail needs reading.

boot/imagemap.c line 2

Quoted 4 lines
boot/imagemap.c +2 -0
@@ -0,0 +1,451 @@
+     ret = device_bind_driver(desc->bdev, "imagemap",
+                              name && *name ? name : "imagemap", &imdev);

Just to check - is the intent that only one imagemap device is ever active per block device? Two calls with name == NULL (or two different partitions selected by index) will bind two children both called imagemap under the same parent, which works but makes the DM tree confusing. How about deriving a unique name (e.g. including the partition number) or documenting the single-instance assumption in imagemap_create()'s kernel-doc ?

boot/imagemap.c line 4

Quoted 6 lines
boot/imagemap.c +4 -0
@@ -0,0 +1,451 @@
+void *imagemap_lookup(struct udevice *dev, loff_t img_offset, ulong size)
+{
+     struct imagemap_priv *priv = dev_get_uclass_priv(dev);
+     const struct imagemap_region *r;

The long block comment is useful but says the second guard prevents underflow on LP64 'where ulong and loff_t have the same rank'. loff_t is signed long long, 64-bit everywhere U-Boot runs; on ILP32 it doesn't share rank with ulong. Please tighten the wording - the guard is needed unconditionally because size is unsigned. Also you could wrap the comment to 80cols so more of it fits on the screen.

Regards,
Simon

#14 lore raw

Re: [PATCH 7/9] configs: sandbox: enable imagemap and its storage backends

In reply to Daniel Golle

Hi Daniel,

Quoted 22 lines · Daniel Golle

configs: sandbox: enable imagemap and its storage backends

Enable CONFIG_IMAGEMAP and, so the end-to-end unit tests can exercise the real block paths, the storage that feeds them: CONFIG_MTD_BLOCK and CONFIG_SPI_FLASH_MTD (a bare non-NAND MTD reached via mtd_blk) and CONFIG_CMD_UBI / CONFIG_UBI_BLOCK (a UBI volume reached via ubi_blk).

Give the sandbox SPI-NOR two device-tree partitions for the mtd_blk test and add a clean sandbox NAND chip (nand@2, no injected errors) so UBI can format on it for the ubiblock test.

arch/sandbox/dts/test.dts | 29 +++++++++++++++++++++++++++++

 configs/sandbox_defconfig |  6 ++++++
 2 files changed, 35 insertions(+)
configs/sandbox_defconfig +0 -0
index 5c97418783f..9e7d734a0b8 100644

configs/sandbox_defconfig CONFIG_UTHREAD=y · line 408

Quoted 8 lines
configs/sandbox_defconfig +5 -0
@@ -400,3 +401,8 @@ CONFIG_UTHREAD=y
 CONFIG_UNIT_TEST=y
 CONFIG_UT_TIME=y
 CONFIG_UT_DM=y
+CONFIG_MTD_BLOCK=y
+CONFIG_SPI_FLASH_MTD=y
+CONFIG_CMD_UBI=y
+CONFIG_MTD_UBI=y
+CONFIG_UBI_BLOCK=y

Tom does this anyway from time to time, but you could run make savedefconfig to get these in the right order - also CONFIG_MTD_UBI is implied by CONFIG_UBI_BLOCK

Quoted 11 lines
arch/sandbox/dts/test.dts +6 -0
@@ -1633,6 +1633,22 @@
                      compatible = "spansion,m25p16", "jedec,spi-nor";
                      spi-max-frequency = <40000000>;
                      sandbox,filename = "spi.bin";
+
+                     partitions {
+                             compatible = "fixed-partitions";
+                             #address-cells = <1>;
+                             #size-cells = <1>;
+

arch/sandbox/dts/test.dts line 1651

Quoted 10 lines
arch/sandbox/dts/test.dts +10 -0
+                             partition@0 {
+                                     label = "nor-fit";
+                                     reg = <0x0 0x100000>;
+                             };
+
+                             partition@100000 {
+                                     label = "nor-spare";
+                                     reg = <0x100000 0x100000>;
+                             };
+                     };

Adding fixed-partitions to spi.bin@0 combined with patch 1 (auto-bind mtd_blk on non-NAND masters) means every sandbox test now grows two extra block-device partitions on this flash. Please check the SPI bootdev tests (bootdev_test_next_prio in test/boot/bootdev.c expects only [email protected] / [email protected]) and dm_test_spi_flash in test/dm/sf.c still pass; if not, the partition table would be better hung off spi.bin@1 (which no existing test walks), or off a dedicated third flash chip.

Quoted 4 lines
arch/sandbox/dts/test.dts +2 -0
@@ -2242,6 +2258,19 @@
+             /* Clean chip (no injected errors) for UBI/ubiblock tests */
+             nand@2 {

arch/sandbox/dts/test.dts line 2269

Quoted 10 lines
arch/sandbox/dts/test.dts +10 -0
+                     reg = <2>;
+                     nand-ecc-mode = "soft";
+                     sandbox,id = [00 e3];
+                     sandbox,erasesize = <(8 * 1024)>;
+                     sandbox,oobsize = <16>;
+                     sandbox,pagesize = <512>;
+                     sandbox,pages = <0x2000>;
+                     sandbox,err-count = <0>;
+                     sandbox,err-step-size = <512>;
+             };

Just to check - sandbox,err-count = <0> is the default, so this and sandbox,err-step-size can be omitted. The comment can go too; the property values already say it.

Regards,
Simon

#16 lore raw

Re: [PATCH 8/9] test: boot: add imagemap unit tests

In reply to Daniel Golle

Hi Daniel,

Quoted 29 lines · Daniel Golle

test: boot: add imagemap unit tests

Add a test suite exercising imagemap:

- the translation-table core (map, lookup, map_to, cleanup, region growth, the block-aligned read path, and edge cases) using a mock RAM-backed reader, so most tests need no real storage;

- a NOR read through mtd_blk: write a known pattern into a device-tree partition of the sandbox SPI-NOR and read it back through imagemap, exercising imagemap -> spl_load_region -> blk_dread -> mtd_blk_read -> mtd_read for real;

- a UBI-volume read through ubiblock: attach UBI on the sandbox NAND, create a volume, write a pattern and read it back through imagemap on the ubi_blk device.

Registered under ut_boot_imagemap in cmd_ut.c.

MAINTAINERS | 1 +

 test/boot/Makefile   |   2 +
 test/boot/imagemap.c | 645 +++++++++++++++++++++++++++++++++++++++++++++++++++
 test/cmd_ut.c        |   2 +
 4 files changed, 650 insertions(+)

test/cmd_ut.c SUITE_DECL(optee); · line 77

Quoted 5 lines
test/cmd_ut.c +1 -0
@@ -73,6 +73,7 @@ SUITE_DECL(optee);
 SUITE_DECL(pci_mps);
 SUITE_DECL(seama);
 SUITE_DECL(setexpr);
+SUITE_DECL(imagemap);
 SUITE_DECL(upl);

Please keep the list alphabetical - imagemap belongs between image_fdt and lib, not between setexpr and upl. Same for the SUITE() entry below.

Quoted 4 lines
test/boot/imagemap.c +2 -0
@@ -0,0 +1,645 @@
+/* Test: map() re-reads when extending a region to a larger size */
+static int imagemap_test_map_extend(struct unit_test_state *uts)

...

test/boot/imagemap.c line 277

Quoted 11 lines
test/boot/imagemap.c +11 -0
+     {
+             void *p3 = imagemap_map(dev, 256, 128);
+             ulong p2_end = map_to_sysmem(p2) + 256;
+
+             ut_assert(!IS_ERR(p3));
+             ut_asserteq(2, priv->regions.count);
+             /* New region must not overlap the extended first region */
+             ut_assert(map_to_sysmem(p3) >= p2_end ||
+                       map_to_sysmem(p3) + 128 <= map_to_sysmem(p2));
+             ut_asserteq_mem(image + 256, p3, 128);
+     }

Please drop the anonymous block - hoist p3/p2_end to the top of the function, or split into its own test case. The nested scope is unusual in U-Boot tests.

Quoted 10 lines
test/boot/imagemap.c +8 -0
@@ -0,0 +1,645 @@
+/* Test: read beyond image size returns error */
+static int imagemap_test_read_oob(struct unit_test_state *uts)
+{
+     struct udevice *dev;
+     u8 image[IMAGE_SIZE];
+     void *p;
+
+     ut_assertok(create_mock_loader(image, IMAGE_SIZE, &dev));

test/boot/imagemap.c line 18

Quoted 10 lines
test/boot/imagemap.c +10 -0
+
+     /* Attempt to map beyond the end of the image */
+     p = imagemap_map(dev, IMAGE_SIZE - 32, 64);
+     ut_assert(IS_ERR(p));
+
+     /* map_to should also fail */
+     u8 dst[64];
+
+     p = imagemap_map_to(dev, IMAGE_SIZE - 32, 64, dst);
+     ut_assert(IS_ERR(p));

Please declare dst at the top of the function - mid-block declarations are still avoided in U-Boot.

test/boot/imagemap.c line 1

Quoted 3 lines
test/boot/imagemap.c +1 -0
@@ -0,0 +1,645 @@
+IMAGEMAP_TEST(imagemap_test_map_basic, 0);

All of these bind and probe devices under dm_root(), and the end-to-end MTD/UBI tests rely on real DM state. Please pass UTF_DM (and UTF_SCAN_FDT for the MTD test that walks device-tree partitions) so DM state is reset between tests - otherwise a test that aborts before imagemap_cleanup() leaks a bound imagemap-test device into the next test and the second run fails on the name clash in device_bind_driver().

Quoted 3 lines
test/boot/imagemap.c +1 -0
@@ -0,0 +1,645 @@
+static int imagemap_test_ubiblock(struct unit_test_state *uts)

...

test/boot/imagemap.c line 614

Quoted 4 lines
test/boot/imagemap.c +4 -0
+     /* Attach UBI by name; this binds the ubi_blk block device */
+     ut_assertok(ubi_part("nand2", NULL));
+     ut_assertok(ubi_create_vol("vol0", 0x10000, true, UBI_VOL_NUM_AUTO,
+                                false));

There is no matching detach, so a second run of the suite (or another UBI-touching test afterwards) sees UBI already attached to nand2 with vol0 present, and ubi_part()/ubi_create_vol() will fail. Please detach UBI at the end (and ideally guard against the pre-attached case at the start) so the test is re-runnable.

Regards,
Simon

#20 lore raw
In reply to Daniel Golle

Hi Daniel,

Quoted 2 lines · Daniel Golle

The first in-tree consumer, a bootmeth for OpenWrt-style FIT firmware, will follow as a separate series.

When that comes, will you be able to add a test for the bootmeth?

Quoted 1 line

boot: fit: support on-demand loading in fit_image_load()

Related: patch 6 adds a struct udevice *imagemap field to both struct bootm_info and struct bootm_headers, but nothing in this series sets either outside the tests. The FIT hook is dead code in the built tree.

Quoted 2 lines

On hardware it has been tested end-to-end on a BananaPi R3 (mt7986), booting a dm-verity-protected rootfs from every storage path the board exposes

Sounds good.

Regards,
Simon

#18 lore raw

Re: [PATCH 9/9] doc: imagemap: document on-demand loading framework

In reply to Daniel Golle

Hi Daniel,

Quoted 7 lines · Daniel Golle

doc: imagemap: document on-demand loading framework

Describe the imagemap framework architecture: the translation table that tracks mapped regions and its phase-determined caching, reading from a block-device partition (including MTD via mtdblock and UBI via ubiblock), the integration with bootm for on-demand FIT sub-image loading, and the SPL build.

The doc doesn't actually describe the SPL build - imagemap is full-U-Boot only. Please either drop that clause from the commit message, or add a short paragraph in imagemap.rst noting the SPL gating (why xpl phases carry no cost).

doc/develop/imagemap.rst line 100

Quoted 11 lines

MAINTAINERS | 1 +

 doc/develop/imagemap.rst | 100 +++++++++++++++++++++++++++++++++++++++++++++++
 doc/develop/index.rst    |   1 +
 3 files changed, 102 insertions(+)
doc/develop/imagemap.rst new file +3 -0
+The ``bootmeth_openwrt`` boot method is the primary consumer of this
+framework, using it to boot OpenWrt-style FIT firmware images stored
+directly on raw storage.

bootmeth_openwrt isn't in-tree yet - so better as 'will be the first in-tree consumer' so the doc doesn't point at a missing symbol.

doc/develop/index.rst Implementation · line 37

Quoted 9 lines
doc/develop/index.rst +1 -0
@@ -31,6 +31,7 @@ Implementation
 
    directories
    bloblist
+   imagemap
    bootstd/index
    ci_testing
    commands

The surrounding entries are alphabetical (bloblist, bootstd, chid, ci_testing, commands, ...) so imagemap belongs later - around global_data/init, not between bloblist and bootstd/index.

doc/develop/imagemap.rst line 3

Quoted 5 lines
doc/develop/imagemap.rst +3 -0
@@ -0,0 +1,100 @@
+``imagemap_create(dev, name, part)``
+   Create an imagemap device over a partition of block device *dev*,
+   selected by *name* or by index *part*.

The real signature also takes a struct udevice **devp out parameter (see include/imagemap.h). Please add it here so the doc matches the API.

doc/develop/imagemap.rst line 11

Quoted 13 lines
doc/develop/imagemap.rst +11 -0
@@ -0,0 +1,100 @@
+Boot Integration
+----------------
+
+The ``imagemap`` framework integrates with the boot pipeline through
+``bootm_run_states()`` in ``boot/bootm.c``:
+
+1. The imagemap device pointer is stored in ``struct bootm_info``
+   and propagated to the global ``images`` structure.
+
+2. ``boot_get_kernel()`` uses ``imagemap_map()`` to read the image
+   header and FIT metadata without loading the full image into RAM.

boot_get_kernel() is a static helper in boot/bootm.c - a doc reader won't easily find it. Please either name the entry point a caller actually reaches (bootm_run_states() / bootm_find_os()) or note that it's internal. Same for the imagemap_map_to() reference under fit_image_load(): it would help to say which caller in patch 6 uses it.

Regards,
Simon

#21 lore raw
In reply to Simon Glass

Hi Simon,

thank you for the extensive review of the series. Meanwhile v2 had already been sent and Tom has already picked it into 'next'. Apart from one issue (see below) I've addressed all correctness issues you have pointed out in a follow up series which I have sent now.

Quoted 8 lines · Simon Glass

Hi Daniel,

Quoted 2 lines · Daniel Golle

The first in-tree consumer, a bootmeth for OpenWrt-style FIT firmware, will follow as a separate series.

When that comes, will you be able to add a test for the bootmeth?

Yes. I've got that planned but it's not done yet. Compared to imagemap and the dm-verity integration (which are both covered by unit tests) the bootmeth by itself also isn't that spectacular ;)

Quoted 5 lines
Quoted 1 line

boot: fit: support on-demand loading in fit_image_load()

Related: patch 6 adds a struct udevice *imagemap field to both struct bootm_info and struct bootm_headers, but nothing in this series sets either outside the tests. The FIT hook is dead code in the built tree.

You are right, that is the gap the bootmeth series is going to close, and until it lands the integration is carried on the strength of out-of-tree testing, which is not a good place for it to sit. I would rather fix that by getting the consumer in than by adding a synthetic caller.

Cheers

Daniel

#22 lore raw

Hi Daniel,

Quoted 7 lines · Daniel Golle

Hi Simon,

thank you for the extensive review of the series. Meanwhile v2 had already been sent and Tom has already picked it into 'next'. Apart from one issue (see below) I've addressed all correctness issues you have pointed out in a follow up series which I have sent now.

Great!

Quoted 21 lines · Simon Glass
Quoted 8 lines

Hi Daniel,

Quoted 2 lines · Daniel Golle

The first in-tree consumer, a bootmeth for OpenWrt-style FIT firmware, will follow as a separate series.

When that comes, will you be able to add a test for the bootmeth?

Yes. I've got that planned but it's not done yet. Compared to imagemap and the dm-verity integration (which are both covered by unit tests) the bootmeth by itself also isn't that spectacular ;)

Quoted 5 lines
Quoted 1 line

boot: fit: support on-demand loading in fit_image_load()

Related: patch 6 adds a struct udevice *imagemap field to both struct bootm_info and struct bootm_headers, but nothing in this series sets either outside the tests. The FIT hook is dead code in the built tree.

Quoted 5 lines

You are right, that is the gap the bootmeth series is going to close, and until it lands the integration is carried on the strength of out-of-tree testing, which is not a good place for it to sit. I would rather fix that by getting the consumer in than by adding a synthetic caller.

Sounds good.

Regards,
Simon