Skip to contentSkip to conversation

imagemap: on-demand FIT loading from storage

PATCH In review Daniel Golle 2 participants 13 messages Aug 25 – Sep 14

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

v2 cover imagemap: on-demand FIT loading from storage

This series adds "imagemap", a small layer for reading FIT images directly from storage on demand instead of copying the whole image into RAM first. It keeps a translation table of the byte ranges already loaded, so a header probe, signature verification and the final load can reuse a single read, and it hooks transparently into fit_image_load() so all existing verification and decompression paths run unchanged for both RAM- and storage-backed images.

An imagemap device is created over a partition of a block device. That covers more than raw disks: an MTD partition is reached through mtdblock and a UBI volume through ubiblock, each exposed as a named block-device partition, so imagemap carries no storage-specific code of its own. NAND bad-block handling and wear-levelling stay with UBI, as they do elsewhere in U-Boot; on NOR a partition is read linearly with end-to-end integrity guaranteed by the FIT hash or signature.

To make those two paths reachable and testable, the first patches fill in the missing block devices: a non-NAND MTD master now gets an mtd_blk device automatically (patch 1), and attaching UBI now binds a ubi_blk device on any flash (patch 3, which makes the SPI-NAND-only eager binding redundant, dropped in patch 4).

The read path is built on the SPL struct spl_load_info abstraction and a new spl_load_region() helper. imagemap is full-U-Boot only; the SPL FIT loader is unaffected.

Testing (sandbox): "ut imagemap" (13 tests) covers the translation-table core plus real end-to-end reads of a NOR partition through mtd_blk and a UBI volume through ubiblock; a CONFIG_IMAGEMAP=n build is unaffected. The whole series passes CI [1].

On hardware it has been tested end-to-end on a BananaPi R3 (mt7986), booting a dm-verity-protected rootfs from every storage path the board
exposes: eMMC and SD (block partition), SPI-NAND (UBI volume via ubiblock) and SPI-NOR (raw MTD partition via mtdblock). The first in-tree consumer, a bootmeth for OpenWrt-style FIT firmware, will follow as a separate series.

Changes in v2 (since v1 [2]):

- Add two prerequisite fixes for the UBI block driver that enabling CONFIG_UBI_BLOCK in sandbox exposes: make it depend on CMD_UBI (its ubi_volume_read()/write() live in cmd/ubi.c, so a CONFIG_CMDLINE=n build otherwise fails to link), and stop the UBI partition scan from claiming any readable block device or dereferencing a NULL ubi_device.
- include/spl.h: express the block-length field directly with #if IS_ENABLED(CONFIG_SPL_LOAD_BLOCK) || CONFIG_IS_ENABLED(IMAGEMAP) instead of a SPL_LOAD_INFO_HAS_BL_LEN helper macro, and fold the note into the existing struct comment (per Tom Rini).
- boot: fit: fix a continuation-line alignment in fit_image_load() flagged by checkpatch (per Tom Rini).

This series began as part of a combined RFC that also carried the OpenWrt boot method and its bootdevs [3]. Following the review there it has been reworked substantially:

- Build on U-Boot's existing SPL external-data loading path rather than a new abstraction: imagemap now uses struct spl_load_info and a new spl_load_region() helper (suggested by Marek Vasut).

- Do not extend the bootm command with a storage source; on-demand loading hooks into fit_image_load() transparently, and the in-tree consumer is a bootstd boot method reached through bootflow/bootdev/bootmeth rather than new bootm syntax (per Tom Rini).

- Move the storage-backed path out of the long fit_image_load() into its own helper, gated with if()/tools_build() instead of an inline #if (per Simon Glass).

- Collapse the three storage-specific backends (block, MTD and UBI) into a single block backend: NOR is read through mtd_blk and SPI-NAND through UBI/ubiblock, so imagemap holds no storage-specific code.

- Send the on-demand loading core on its own; the OpenWrt boot method, the MTD/UBI bootdevs and board enablement follow as separate series once this lands.

A note on that follow-up: to tell Linux which FIT it booted, U-Boot uses the install-uuid mechanism of [4], which [5] could supersede. Both have stalled.

[1] https://git.u-boot-project.org/u-boot/contributors/dgolle/u-boot/-/pipelines/1067
[2] https://lore.kernel.org/u-boot/[email protected]/
[3] https://lore.kernel.org/u-boot/[email protected]/
[4] https://github.com/open-source-firmware/flat-image-tree/pull/40
[5] https://github.com/open-source-firmware/flat-image-tree/pull/44

Daniel Golle (11):
mtd: bind an mtd_blk device for non-NAND MTD masters
mtd: nand: sandbox: forbid sub-page writes
cmd: ubi: create a ubi_blk device when attaching UBI
mtd: nand: spi: drop the eager ubi_blk binding
mtd: ubi: block: build only with CONFIG_CMD_UBI
mtd: ubi: block: only claim UBI block devices in the partition scan
boot: add imagemap on-demand loading from storage
boot: fit: support on-demand loading in fit_image_load()
configs: sandbox: enable imagemap and its storage backends
test: boot: add imagemap unit tests
doc: imagemap: document on-demand loading framework

 MAINTAINERS                      |   9 +
 arch/sandbox/dts/test.dts        |  29 ++
 boot/Kconfig                     |  16 +
 boot/Makefile                    |   2 +
 boot/bootm.c                     |  70 +++-
 boot/image-fit.c                 | 109 +++++-
 boot/imagemap.c                  | 451 +++++++++++++++++++++
 cmd/ubi.c                        |  27 ++
 configs/sandbox_defconfig        |   6 +
 doc/develop/imagemap.rst         | 100 +++++
 doc/develop/index.rst            |   1 +
 drivers/mtd/mtdcore.c            |  24 ++
 drivers/mtd/nand/raw/sand_nand.c |   2 +
 drivers/mtd/nand/spi/core.c      |   3 -
 drivers/mtd/ubi/Kconfig          |   1 +
 drivers/mtd/ubi/part.c           |   9 +
 include/bootm.h                  |   2 +
 include/dm/uclass-id.h           |   1 +
 include/image.h                  |   2 +
 include/imagemap.h               | 136 +++++++
 include/spl.h                    |  38 +-
 test/boot/Makefile               |   2 +
 test/boot/imagemap.c             | 645 +++++++++++++++++++++++++++++++
 test/cmd_ut.c                    |   2 +
 24 files changed, 1671 insertions(+), 16 deletions(-)
 create mode 100644 boot/imagemap.c
 create mode 100644 doc/develop/imagemap.rst
 create mode 100644 include/imagemap.h
 create mode 100644 test/boot/imagemap.c

base-commit: 8841b7b292c74864acf4ed0bd8ce05839252c1f2

Signature · 1 line
2.55.0
#1 lore raw

PATCH v2 1/11 mtd: bind an mtd_blk device for non-NAND MTD masters

Only the SPI-NAND bind path currently creates an mtd_blk block device, so a bare NOR (parallel NOR, SPI-NOR, ...) has no block-device access at all. Bind an mtd_blk device for every non-NAND MTD master when CONFIG_MTD_BLOCK is enabled, so its partitions become reachable through the block layer (and thus by bootstd, imagemap, ...).

NAND is excluded on purpose: it needs UBI, which brings its own block device. Partitions inherit the master's block device via the "mtd" partition type, so only masters are bound.

mtd_bind() keeps a pointer to the passed mtd_info pointer (SPI-NAND fills it after bind), so the generic caller provides a small heap slot that lives as long as the block device; an MTD master is never removed in practice.

---
 drivers/mtd/mtdcore.c | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)
drivers/mtd/mtdcore.c +24 -0
index 3bfa5aebbc6..e0d0b5ae2e9 100644
@@ -494,6 +494,30 @@ int add_mtd_device(struct mtd_info *mtd)
 #endif
 
 	mutex_unlock(&mtd_table_mutex);
+
+	if (IS_ENABLED(CONFIG_MTD_BLOCK) && mtd->dev &&
+	    !mtd_is_partition(mtd) && !mtd_type_is_nand(mtd)) {
+		/*
+		 * Expose a non-NAND MTD master (parallel NOR, SPI-NOR, ...) as
+		 * a mtd_blk block device, so its partitions are reachable
+		 * through the block layer (and thus by imagemap, bootstd, ...).
+		 * NAND is excluded: it needs UBI, which brings its own block
+		 * device. Partitions inherit the master's block device via the
+		 * "mtd" partition type, so only masters are bound.
+		 *
+		 * mtd_bind() keeps a pointer to the passed mtd_info pointer, so
+		 * it needs storage that lives as long as the block device; the
+		 * MTD master is never removed in practice, so a small heap slot
+		 * is fine.
+		 */
+		struct mtd_info **mtdp = kmalloc(sizeof(*mtdp), GFP_KERNEL);
+
+		if (mtdp) {
+			*mtdp = mtd;
+			mtd_bind(mtd->dev, mtdp);
+		}
+	}
+
 	/* We _know_ we aren't being removed, because
 	   our caller is still holding us here. So none
 	   of this try_ nonsense, and no bitching about it
Signature · 1 line
2.55.0
#2 lore raw

PATCH v2 2/11 mtd: nand: sandbox: forbid sub-page writes

In reply to Daniel Golle

The sandbox NAND model programs each page in a single operation and rejects a second write to an already-programmed page. UBI writes the EC and VID headers into the first page in two separate sub-page operations, which the model fails with -EIO, so UBI cannot attach to a sandbox NAND.

Advertise NAND_NO_SUBPAGE_WRITE so callers place each header in its own page, matching what the model can actually do. This lets the imagemap UBI-volume unit test attach UBI on the sandbox NAND.

---
 drivers/mtd/nand/raw/sand_nand.c | 2 ++
 1 file changed, 2 insertions(+)
drivers/mtd/nand/raw/sand_nand.c +2 -0
index 3678bb8a41b..5c48b43a882 100644
@@ -602,6 +602,8 @@ static int sand_nand_probe(struct udevice *dev)
 
 		nand = &chip->nand;
 		nand->options = not_xpl() ? 0 : NAND_SKIP_BBTSCAN;
+		/* Each page is programmed in a single operation only */
+		nand->options |= NAND_NO_SUBPAGE_WRITE;
 		nand->flash_node = np;
 		nand->dev_ready = sand_nand_dev_ready;
 		nand->cmdfunc = sand_nand_command;
Signature · 1 line
2.55.0
#3 lore raw

PATCH v2 3/11 cmd: ubi: create a ubi_blk device when attaching UBI

In reply to Daniel Golle

A UBI volume can be reached as a partition on the ubi_blk block device, but that device is only bound from spinand_bind(), so it is missing on every other flash type (raw NAND, SPI-NOR, ...).

Bind a ubi_blk device when a UBI partition is attached, unless one has already been created, parented to the MTD device UBI sits on. UBI volumes then become reachable as block-device partitions regardless of the underlying flash. The imagemap block path relies on this to read a UBI volume.

---
 cmd/ubi.c | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)
cmd/ubi.c +27 -0
index 2b206141a21..91ba21504f4 100644
@@ -20,7 +20,10 @@
 #include <mtd.h>
 #include <nand.h>
 #include <onenand_uboot.h>
+#include <dm/device.h>
 #include <dm/devres.h>
+#include <dm/root.h>
+#include <dm/uclass-internal.h>
 #include <linux/mtd/mtd.h>
 #include <linux/mtd/partitions.h>
 #include <linux/err.h>
@@ -670,6 +673,29 @@ int ubi_detach(void)
 	return 0;
 }
 
+#if CONFIG_IS_ENABLED(UBI_BLOCK)
+static void ubi_blk_bind_once(void)
+{
+	struct udevice *dev;
+
+	/*
+	 * A single ubi_blk device serves all volumes of the attached UBI
+	 * device, the volume being selected through the block descriptor's
+	 * hwpart. Bind one when UBI is attached -- unless a previous attach
+	 * already did -- parented to the MTD device UBI sits on.
+	 */
+	for (uclass_find_first_device(UCLASS_BLK, &dev); dev;
+	     uclass_find_next_device(&dev)) {
+		if (dev->driver == DM_DRIVER_GET(ubi_blk))
+			return;
+	}
+
+	ubi_bind(ubi && ubi->mtd && ubi->mtd->dev ? ubi->mtd->dev : dm_root());
+}
+#else
+static inline void ubi_blk_bind_once(void) { }
+#endif
+
 int ubi_part(const char *part_name, const char *vid_header_offset)
 {
 	struct mtd_info *mtd;
@@ -698,6 +724,7 @@ int ubi_part(const char *part_name, const char *vid_header_offset)
 	}
 
 	ubi = ubi_devices[0];
+	ubi_blk_bind_once();
 
 	return 0;
 }
Signature · 1 line
2.55.0
#4 lore raw

PATCH v2 4/11 mtd: nand: spi: drop the eager ubi_blk binding

In reply to Daniel Golle

spinand_bind() binds a ubi_blk device at bind time. Now that "cmd:
ubi: create a ubi_blk device when attaching UBI" binds one generically when UBI is attached -- with an identical result, only lazily and for any flash type -- the SPI-NAND-specific eager call is redundant.

Drop it to avoid the duplicate code path (a ubi_blk is unusable before UBI is attached anyway). The mtd_blk binding is kept: bare mtd_blk access to SPI-NAND remains available for users who want it.

---
 drivers/mtd/nand/spi/core.c | 3 ---
 1 file changed, 3 deletions(-)
drivers/mtd/nand/spi/core.c +0 -3
index 14af4264612..fb8ca0a787d 100644
@@ -1654,9 +1654,6 @@ static int spinand_bind(struct udevice *dev)
 			if (ret)
 				return ret;
 		}
-
-		if (CONFIG_IS_ENABLED(UBI_BLOCK))
-			return ubi_bind(dev);
 	}
 
 	return 0;
Signature · 1 line
2.55.0
#5 lore raw

PATCH v2 5/11 mtd: ubi: block: build only with CONFIG_CMD_UBI

In reply to Daniel Golle

The UBI block driver calls ubi_volume_read() and ubi_volume_write(), which are defined in cmd/ubi.c and thus only built with CONFIG_CMD_UBI. Selecting UBI_BLOCK without CMD_UBI, for example in a CONFIG_CMDLINE=n build, fails to link:

drivers/mtd/ubi/block.c: undefined reference to ubi_volume_read' drivers/mtd/ubi/block.c: undefined reference to ubi_volume_write'

Make the dependency explicit so UBI_BLOCK is only offered when CMD_UBI is enabled. The only in-tree user, topic_miami_defconfig, already enables CMD_UBI, so no board is affected.

Fixes 9daad11ad17 ("drivers: introduce UBI block abstraction")
---
 drivers/mtd/ubi/Kconfig | 1 +
 1 file changed, 1 insertion(+)
drivers/mtd/ubi/Kconfig +1 -0
index e523a4c4707..bde8664fe80 100644
@@ -116,6 +116,7 @@ config MTD_UBI_FM_DEBUG
 
 config UBI_BLOCK
 	bool "Enable UBI block device support"
+	depends on CMD_UBI
 	select BLK
 	help
 	  Enable UBI block device support using blk_ops abstraction.
Signature · 1 line
2.55.0
#6 lore raw

PATCH v2 6/11 mtd: ubi: block: only claim UBI block devices in the partition scan

In reply to Daniel Golle

part_test_ubi() reads the first block and returns success for any readable device, so a bare block device with no partition table (for example a sandbox "host bind" image) is mis-detected as UBI once CONFIG_UBI_BLOCK is enabled. part_get_info_ubi() then calls ubi_get_volume_by_index(), which dereferences ubi_devices[0] without a NULL check and crashes when no UBI device is attached.

Fail part_test_ubi() when no UBI device is attached, and guard the volume lookup and print helpers against a NULL ubi_device, so the driver only claims devices while UBI is present and never dereferences a missing one.

Fixes aa5b67ce226 ("disk: support UBI partitions")
---
 drivers/mtd/ubi/part.c | 9 +++++++++
 1 file changed, 9 insertions(+)
drivers/mtd/ubi/part.c +9 -0
index 6c017eb7299..e1bdd9e3008 100644
@@ -19,6 +19,9 @@ static struct ubi_volume *ubi_get_volume_by_index(int vol_id)
 	struct ubi_device *ubi = get_ubi_device();
 	int i;
 
+	if (!ubi)
+		return NULL;
+
 	for (i = 0; i < (ubi->vtbl_slots + 1); i++) {
 		struct ubi_volume *volume = ubi->volumes[i];
 
@@ -66,6 +69,9 @@ static void __maybe_unused part_print_ubi(struct blk_desc *dev_desc)
 	struct ubi_device *ubi = get_ubi_device();
 	int i;
 
+	if (!ubi)
+		return;
+
 	for (i = 0; i < (ubi->vtbl_slots + 1); i++) {
 		struct ubi_volume *volume = ubi->volumes[i];
 
@@ -83,6 +89,9 @@ static int part_test_ubi(struct blk_desc *dev_desc)
 {
 	ALLOC_CACHE_ALIGN_BUFFER(unsigned char, buffer, dev_desc->blksz);
 
+	if (!get_ubi_device())
+		return -1;
+
 	if (blk_dread(dev_desc, 0, 1, (ulong *)buffer) != 1)
 		return -1;
 
Signature · 1 line
2.55.0
#7 lore raw

PATCH v2 7/11 boot: add imagemap on-demand loading from storage

In reply to Daniel Golle

Introduce imagemap, a small layer that reads image data from a storage device on demand and keeps a translation table of the byte ranges already loaded into RAM, instead of copying the whole image up front.

An imagemap device (UCLASS_IMAGEMAP) is created over a partition of a block device with imagemap_create(); imagemap_map() then returns a RAM pointer for any byte range, reading it in on first access and reusing it on later accesses -- so a header probe, verification and the final load of the same range share a single read. Block devices cover more than raw disks: an MTD partition is reached through mtdblock and a UBI volume through ubiblock, each exposed as a named block-device partition, so imagemap needs no storage-specific code of its own.

The read path is built on the SPL struct spl_load_info abstraction and a new spl_load_region() helper (added to spl.h): the block reader reports the device block length and spl_load_region() performs the native sector alignment. imagemap_map_to() keeps a payload byte-exact at a caller's load address, reading the aligned middle straight to the destination and bouncing only the partial head/tail block.

Regions that are not placed at a caller-supplied address are allocated through the LMB allocator and released on cleanup; the translation table doubles as the registry of those allocations. imagemap is gated to the full-U-Boot phase, so SPL and TPL carry no new cost.

---
 MAINTAINERS            |   7 +
 boot/Kconfig           |  16 ++
 boot/Makefile          |   2 +
 boot/imagemap.c        | 451 +++++++++++++++++++++++++++++++++++++++++
 include/dm/uclass-id.h |   1 +
 include/imagemap.h     | 136 +++++++++++++
 include/spl.h          |  38 +++-
 7 files changed, 647 insertions(+), 4 deletions(-)
 create mode 100644 boot/imagemap.c
 create mode 100644 include/imagemap.h
MAINTAINERS +7 -0
index acaba95ed03..48ee01e033d 100644
@@ -1186,6 +1186,13 @@ S:	Maintained
 F:	drivers/timer/goldfish_timer.c
 F:	include/goldfish_timer.h
 
+IMAGEMAP
+M:	Daniel Golle <[email protected]>
+L:	[email protected]
+S:	Maintained
+F:	boot/imagemap.c
+F:	include/imagemap.h
+
 INTERCONNECT
 M:	Neil Armstrong <[email protected]>
 S:	Maintained
boot/Kconfig +16 -0
index c67dc0ba493..0d060aa25e4 100644
@@ -1205,6 +1205,22 @@ config SYS_BOOT_RAMDISK_HIGH
 
 endmenu		# Boot images
 
+config IMAGEMAP
+	bool "On-demand image loading from storage"
+	depends on DM && LMB && BLK && PARTITIONS
+	help
+	  Read image data from a storage device on demand instead of
+	  copying the whole image into RAM first. A translation table maps
+	  already-loaded byte ranges to their RAM addresses to avoid
+	  redundant reads, and scratch allocations are managed through the
+	  LMB allocator.
+
+	  Images are read from a partition on a block device, identified by
+	  number or name. MTD partitions (via mtdblock) and UBI volumes
+	  (via ubiblock) are reached as block-device partitions too. Used
+	  by bootm when a storage device is specified instead of a RAM
+	  address.
+
 config DISTRO_DEFAULTS
 	bool "(deprecated) Script-based booting of Linux distributions"
 	select CMDLINE
boot/Makefile +2 -0
index 7fb56e7ef37..b4200ebab3d 100644
@@ -73,6 +73,8 @@ obj-$(CONFIG_$(PHASE_)BOOTMETH_VBE_SIMPLE_OS) += vbe_simple_os.o
 
 obj-$(CONFIG_$(PHASE_)BOOTMETH_ANDROID) += bootmeth_android.o
 
+obj-$(CONFIG_$(PHASE_)IMAGEMAP) += imagemap.o
+
 obj-$(CONFIG_$(PHASE_)BOOTMETH_VBE_ABREC) += vbe_abrec.o vbe_common.o
 obj-$(CONFIG_$(PHASE_)BOOTMETH_VBE_ABREC_FW) += vbe_abrec_fw.o
 obj-$(CONFIG_$(PHASE_)BOOTMETH_VBE_ABREC_OS) += vbe_abrec_os.o
boot/imagemap.c new file +451 -0

451 lines not on this page · raw message

include/dm/uclass-id.h +1 -0
index fe0aae2720c..7de9312cde7 100644
@@ -75,6 +75,7 @@ enum uclass_id {
 	UCLASS_HASH,		/* Hash device */
 	UCLASS_HWSPINLOCK,	/* Hardware semaphores */
 	UCLASS_HOST,		/* Sandbox host device */
+	UCLASS_IMAGEMAP,	/* On-demand image loading from storage */
 	UCLASS_I2C,		/* I2C bus */
 	UCLASS_I2C_EEPROM,	/* I2C EEPROM device */
 	UCLASS_I2C_GENERIC,	/* Generic I2C device */
include/imagemap.h new file +136 -0

136 lines not on this page · raw message

include/spl.h +34 -4
index 5078d7525ab..8a9f799f192 100644
@@ -346,14 +346,14 @@ typedef ulong (*spl_load_reader)(struct spl_load_info *load, ulong sector,
  *
  * @read: Function to call to read from the device
  * @priv: Private data for the device
- * @bl_len: Block length for reading in bytes
+ * @bl_len: Block length in bytes (CONFIG_SPL_LOAD_BLOCK, CONFIG_IMAGEMAP)
  * @phase: Image phase to load
  * @no_fdt_update: true to update the FDT with any loadables that are loaded
  */
 struct spl_load_info {
 	spl_load_reader read;
 	void *priv;
-#if IS_ENABLED(CONFIG_SPL_LOAD_BLOCK)
+#if IS_ENABLED(CONFIG_SPL_LOAD_BLOCK) || CONFIG_IS_ENABLED(IMAGEMAP)
 	u16 bl_len;
 #endif
 #if CONFIG_IS_ENABLED(BOOTMETH_VBE)
@@ -364,7 +364,7 @@ struct spl_load_info {
 
 static inline int spl_get_bl_len(struct spl_load_info *info)
 {
-#if IS_ENABLED(CONFIG_SPL_LOAD_BLOCK)
+#if IS_ENABLED(CONFIG_SPL_LOAD_BLOCK) || CONFIG_IS_ENABLED(IMAGEMAP)
 	return info->bl_len;
 #else
 	return 1;
@@ -373,7 +373,7 @@ static inline int spl_get_bl_len(struct spl_load_info *info)
 
 static inline void spl_set_bl_len(struct spl_load_info *info, int bl_len)
 {
-#if IS_ENABLED(CONFIG_SPL_LOAD_BLOCK)
+#if IS_ENABLED(CONFIG_SPL_LOAD_BLOCK) || CONFIG_IS_ENABLED(IMAGEMAP)
 	info->bl_len = bl_len;
 #else
 	if (bl_len != 1)
@@ -429,6 +429,36 @@ static inline void spl_load_init(struct spl_load_info *load,
 	xpl_set_fdt_update(load, true);
 }
 
+/**
+ * spl_load_region() - Read a block-aligned region into a buffer
+ *
+ * Reads @size bytes starting at @offset from the device described by @info
+ * into @dst, rounding the read down/up to the device block length so that
+ * block media are read on their native boundaries. On byte-addressed media
+ * (bl_len == 1) every alignment folds to identity.
+ *
+ * @info:	information about the device to read from
+ * @offset:	byte offset on the device of the first wanted byte
+ * @size:	number of wanted bytes
+ * @dst:	buffer to read the block-aligned region into
+ * Return:	on success, the number of leading padding bytes in @dst that
+ *		precede the wanted data (0 on byte media); a negative error
+ *		number on failure.
+ */
+static inline long spl_load_region(struct spl_load_info *info, loff_t offset,
+				   ulong size, void *dst)
+{
+	ulong bl_len = spl_get_bl_len(info);
+	ulong overhead = offset & (bl_len - 1);
+	loff_t roff = ALIGN_DOWN(offset, bl_len);
+	ulong rsize = ALIGN(size + overhead, bl_len);
+
+	if (info->read(info, roff, rsize, dst) < rsize)
+		return -EIO;
+
+	return overhead;
+}
+
 /*
  * We need to know the position of U-Boot in memory so we can jump to it. We
  * allow any U-Boot binary to be used (u-boot.bin, u-boot-nodtb.bin,
Signature · 1 line
2.55.0
#8 lore raw

PATCH v2 8/11 boot: fit: support on-demand loading in fit_image_load()

In reply to Daniel Golle

Integrate storage-backed image loading transparently into the existing FIT flow so that all verification, decompression, and copy-to-load- address code runs unchanged for both RAM-backed and storage-backed images.

Hook imagemap into fit_image_get_data(): when gd->imagemap is set and the sub-image uses external data, call imagemap_map() to bring the payload into RAM instead of returning the unmapped fit + offset pointer. Every existing caller, including fit_image_verify() called from fit_image_select(), transparently gets valid data without any changes to the authentication path.

For uncompressed sub-images with a known load address, pre-populate the translation table via imagemap_map_to() before verification runs. This loads data directly to the final RAM destination; when fit_image_get_data() is later called, the translation table hit returns the same pointer, achieving zero-copy. Compressed images or those without a load address fall through to the lazy imagemap_map() path.

Filesystem sub-images protected by dm-verity are returned early and stay on storage, as the kernel verifies them at block level.

---
 boot/bootm.c     |  70 +++++++++++++++++++++++++++---
 boot/image-fit.c | 109 ++++++++++++++++++++++++++++++++++++++++++++++-
 include/bootm.h  |   2 +
 include/image.h  |   2 +
 4 files changed, 174 insertions(+), 9 deletions(-)
boot/bootm.c +63 -7
index 3bce8586834..22b0394599a 100644
@@ -14,6 +14,7 @@
 #include <env.h>
 #include <errno.h>
 #include <fdt_support.h>
+#include <imagemap.h>
 #include <irq_func.h>
 #include <lmb.h>
 #include <log.h>
@@ -147,7 +148,22 @@ static int boot_get_kernel(const char *addr_fit, struct bootm_headers *images,
 
 	/* check image type, for FIT images get FIT kernel node */
 	*os_data = *os_len = 0;
-	buf = map_sysmem(img_addr, 0);
+	if (IS_ENABLED(CONFIG_IMAGEMAP) && images->imagemap) {
+		/*
+		 * Storage path: read enough bytes to detect the image
+		 * format. genimg_get_kernel_addr_fit() above still
+		 * parsed any #config / :subimage suffix so the FIT
+		 * selection variables are populated.
+		 */
+		buf = imagemap_map(images->imagemap, 0, 64);
+		if (IS_ERR(buf)) {
+			puts("Cannot read image header from storage\n");
+			return PTR_ERR(buf);
+		}
+		img_addr = map_to_sysmem(buf);
+	} else {
+		buf = map_sysmem(img_addr, 0);
+	}
 	switch (genimg_get_format(buf)) {
 #if CONFIG_IS_ENABLED(LEGACY_IMAGE_FORMAT)
 	case IMAGE_FORMAT_LEGACY:
@@ -193,6 +209,20 @@ static int boot_get_kernel(const char *addr_fit, struct bootm_headers *images,
 #endif
 #if CONFIG_IS_ENABLED(FIT)
 	case IMAGE_FORMAT_FIT:
+		if (IS_ENABLED(CONFIG_IMAGEMAP) && images->imagemap) {
+			/*
+			 * Extend the mapping to cover the full FIT
+			 * FDT structure so all metadata is accessible.
+			 */
+			size_t fdt_sz = fdt_totalsize(buf);
+
+			buf = imagemap_map(images->imagemap, 0, fdt_sz);
+			if (IS_ERR(buf)) {
+				puts("Cannot read FIT header from storage\n");
+				return PTR_ERR(buf);
+			}
+			img_addr = map_to_sysmem(buf);
+		}
 		os_noffset = fit_image_load(images, img_addr,
 				&fit_uname_kernel, &fit_uname_config,
 				IH_ARCH_DEFAULT, IH_TYPE_KERNEL,
@@ -1033,11 +1063,21 @@ int bootm_run_states(struct bootm_info *bmi, int states)
 	 * Work through the states and see how far we get. We stop on
 	 * any error.
 	 */
-	if (states & BOOTM_STATE_START)
+	if (states & BOOTM_STATE_START) {
 		ret = bootm_start();
+		/*
+		 * bootm_start() zeroes the global images struct. Restore
+		 * the loader pointer so the storage-backed path works.
+		 */
+		if (IS_ENABLED(CONFIG_IMAGEMAP) && bmi->imagemap)
+			images->imagemap = bmi->imagemap;
+	}
 
-	if (!ret && (states & BOOTM_STATE_PRE_LOAD))
-		ret = bootm_pre_load(bmi->addr_img);
+	if (!ret && (states & BOOTM_STATE_PRE_LOAD)) {
+		/* Pre-load verification is not applicable to storage boot */
+		if (!IS_ENABLED(CONFIG_IMAGEMAP) || !images->imagemap)
+			ret = bootm_pre_load(bmi->addr_img);
+	}
 
 	if (!ret && (states & BOOTM_STATE_FINDOS))
 		ret = bootm_find_os(bmi->cmd_name, bmi->addr_img);
@@ -1045,8 +1085,11 @@ int bootm_run_states(struct bootm_info *bmi, int states)
 	if (!ret && (states & BOOTM_STATE_FINDOTHER)) {
 		ulong img_addr;
 
-		img_addr = bmi->addr_img ? hextoul(bmi->addr_img, NULL)
-			: image_load_addr;
+		if (IS_ENABLED(CONFIG_IMAGEMAP) && images->imagemap)
+			img_addr = images->os.start;
+		else
+			img_addr = bmi->addr_img ? hextoul(bmi->addr_img, NULL)
+				: image_load_addr;
 		ret = bootm_find_other(img_addr, bmi->conf_ramdisk,
 				       bmi->conf_fdt);
 	}
@@ -1145,11 +1188,24 @@ int bootm_run_states(struct bootm_info *bmi, int states)
 	}
 
 	/* Now run the OS! We hope this doesn't return */
-	if (!ret && (states & BOOTM_STATE_OS_GO))
+	if (!ret && (states & BOOTM_STATE_OS_GO)) {
+		/* Release storage backend before jumping - no return expected */
+		if (IS_ENABLED(CONFIG_IMAGEMAP) && images->imagemap) {
+			imagemap_cleanup(images->imagemap);
+			images->imagemap = NULL;
+		}
+
 		ret = boot_selected_os(BOOTM_STATE_OS_GO, bmi, boot_fn);
+	}
 
 	/* Deal with any fallout */
 err:
+	/* Clean up imagemap on error (not reached on successful boot) */
+	if (IS_ENABLED(CONFIG_IMAGEMAP) && images->imagemap) {
+		imagemap_cleanup(images->imagemap);
+		images->imagemap = NULL;
+	}
+
 	if (iflag)
 		enable_interrupts();
 
boot/image-fit.c +107 -2

139 lines not on this page · raw message

include/bootm.h +2 -0
index f6958be751a..d1aac3d44df 100644
@@ -40,6 +40,7 @@ struct cmd_tbl;
  *	boot_get_fdt() for processing, or NULL for none
  * @boot_progress: true to show boot progress
  * @images: images information
+ * @imagemap: imagemap device for storage-backed boot (NULL for in-memory)
  * @cmd_name: command which invoked this operation, e.g. "bootm"
  * @argc: Number of arguments to the command (excluding the actual command).
  *	This is 0 if there are no arguments
@@ -51,6 +52,7 @@ struct bootm_info {
 	const char *conf_fdt;
 	bool boot_progress;
 	struct bootm_headers *images;
+	struct udevice *imagemap;
 	const char *cmd_name;
 	int argc;
 	char *const *argv;
include/image.h +2 -0
index 6edcb1995bf..f5baec9a516 100644
@@ -412,6 +412,8 @@ struct bootm_headers {
 
 	int		verify;		/* env_get("verify")[0] != 'n' */
 
+	struct udevice *imagemap;	/* on-demand storage loader, or NULL */
+
 #define BOOTM_STATE_START	0x00000001
 #define BOOTM_STATE_FINDOS	0x00000002
 #define BOOTM_STATE_FINDOTHER	0x00000004
Signature · 1 line
2.55.0
#9 lore raw

PATCH v2 9/11 configs: sandbox: enable imagemap and its storage backends

In reply to Daniel Golle

Enable CONFIG_IMAGEMAP and, so the end-to-end unit tests can exercise the real block paths, the storage that feeds them: CONFIG_MTD_BLOCK and CONFIG_SPI_FLASH_MTD (a bare non-NAND MTD reached via mtd_blk) and CONFIG_CMD_UBI / CONFIG_UBI_BLOCK (a UBI volume reached via ubi_blk).

Give the sandbox SPI-NOR two device-tree partitions for the mtd_blk test and add a clean sandbox NAND chip (nand@2, no injected errors) so UBI can format on it for the ubiblock test.

---
 arch/sandbox/dts/test.dts | 29 +++++++++++++++++++++++++++++
 configs/sandbox_defconfig |  6 ++++++
 2 files changed, 35 insertions(+)
arch/sandbox/dts/test.dts +29 -0
index bed2a4b65ab..b794a44a4e1 100644
@@ -1623,6 +1623,22 @@
 			compatible = "spansion,m25p16", "jedec,spi-nor";
 			spi-max-frequency = <40000000>;
 			sandbox,filename = "spi.bin";
+
+			partitions {
+				compatible = "fixed-partitions";
+				#address-cells = <1>;
+				#size-cells = <1>;
+
+				partition@0 {
+					label = "nor-fit";
+					reg = <0x0 0x100000>;
+				};
+
+				partition@100000 {
+					label = "nor-spare";
+					reg = <0x100000 0x100000>;
+				};
+			};
 		};
 		spi.bin@1 {
 			reg = <1>;
@@ -2232,6 +2248,19 @@
 			sandbox,err-count = <3>;
 			sandbox,err-step-size = <512>;
 		};
+
+		/* Clean chip (no injected errors) for UBI/ubiblock tests */
+		nand@2 {
+			reg = <2>;
+			nand-ecc-mode = "soft";
+			sandbox,id = [00 e3];
+			sandbox,erasesize = <(8 * 1024)>;
+			sandbox,oobsize = <16>;
+			sandbox,pagesize = <512>;
+			sandbox,pages = <0x2000>;
+			sandbox,err-count = <0>;
+			sandbox,err-step-size = <512>;
+		};
 	};
 
 	graph1 {
configs/sandbox_defconfig +6 -0
index f65e50590a3..05876cb36ba 100644
@@ -30,6 +30,7 @@ CONFIG_BOOTMETH_RAUC=y
 CONFIG_UPL=y
 CONFIG_LEGACY_IMAGE_FORMAT=y
 CONFIG_MEASURED_BOOT=y
+CONFIG_IMAGEMAP=y
 CONFIG_BOOTSTAGE=y
 CONFIG_BOOTSTAGE_REPORT=y
 CONFIG_BOOTSTAGE_FDT=y
@@ -399,3 +400,8 @@ CONFIG_UTHREAD=y
 CONFIG_UNIT_TEST=y
 CONFIG_UT_TIME=y
 CONFIG_UT_DM=y
+CONFIG_MTD_BLOCK=y
+CONFIG_SPI_FLASH_MTD=y
+CONFIG_CMD_UBI=y
+CONFIG_MTD_UBI=y
+CONFIG_UBI_BLOCK=y
Signature · 1 line
2.55.0
#10 lore raw

PATCH v2 10/11 test: boot: add imagemap unit tests

In reply to Daniel Golle

Add a test suite exercising imagemap:
- the translation-table core (map, lookup, map_to, cleanup, region growth, the block-aligned read path, and edge cases) using a mock RAM-backed reader, so most tests need no real storage;

- a NOR read through mtd_blk: write a known pattern into a device-tree partition of the sandbox SPI-NOR and read it back through imagemap, exercising imagemap -> spl_load_region -> blk_dread -> mtd_blk_read -> mtd_read for real;

- a UBI-volume read through ubiblock: attach UBI on the sandbox NAND, create a volume, write a pattern and read it back through imagemap on the ubi_blk device.

Registered under ut_boot_imagemap in cmd_ut.c.

---
 MAINTAINERS          |   1 +
 test/boot/Makefile   |   2 +
 test/boot/imagemap.c | 645 +++++++++++++++++++++++++++++++++++++++++++
 test/cmd_ut.c        |   2 +
 4 files changed, 650 insertions(+)
 create mode 100644 test/boot/imagemap.c
MAINTAINERS +1 -0
index 48ee01e033d..e8f0e94597f 100644
@@ -1192,6 +1192,7 @@ L:	[email protected]
 S:	Maintained
 F:	boot/imagemap.c
 F:	include/imagemap.h
+F:	test/boot/imagemap.c
 
 INTERCONNECT
 M:	Neil Armstrong <[email protected]>
test/boot/Makefile +2 -0
index 59a87028704..8dbecabde79 100644
@@ -27,3 +27,5 @@ endif
 obj-$(CONFIG_BOOTMETH_VBE) += vbe_fixup.o
 
 obj-$(CONFIG_UPL) += upl.o
+
+obj-$(CONFIG_IMAGEMAP) += imagemap.o
test/boot/imagemap.c new file +645 -0

645 lines not on this page · raw message

test/cmd_ut.c +2 -0
index 4328670d0d6..17610cdc2af 100644
@@ -73,6 +73,7 @@ SUITE_DECL(optee);
 SUITE_DECL(pci_mps);
 SUITE_DECL(seama);
 SUITE_DECL(setexpr);
+SUITE_DECL(imagemap);
 SUITE_DECL(upl);
 
 static struct suite suites[] = {
@@ -102,6 +103,7 @@ static struct suite suites[] = {
 	SUITE(pci_mps, "PCI Express Maximum Payload Size"),
 	SUITE(seama, "seama command parameters loading and decoding"),
 	SUITE(setexpr, "setexpr command"),
+	SUITE(imagemap, "imagemap on-demand storage loading"),
 	SUITE(upl, "Universal payload support"),
 };
 
Signature · 1 line
2.55.0
#11 lore raw

PATCH v2 11/11 doc: imagemap: document on-demand loading framework

In reply to Daniel Golle

Describe the imagemap framework architecture: the translation table that tracks mapped regions and its phase-determined caching, reading from a block-device partition (including MTD via mtdblock and UBI via ubiblock), the integration with bootm for on-demand FIT sub-image loading, and the SPL build.

---
 MAINTAINERS              |   1 +
 doc/develop/imagemap.rst | 100 +++++++++++++++++++++++++++++++++++++++
 doc/develop/index.rst    |   1 +
 3 files changed, 102 insertions(+)
 create mode 100644 doc/develop/imagemap.rst
MAINTAINERS +1 -0
index e8f0e94597f..f874443d87c 100644
@@ -1191,6 +1191,7 @@ M:	Daniel Golle <[email protected]>
 L:	[email protected]
 S:	Maintained
 F:	boot/imagemap.c
+F:	doc/develop/imagemap.rst
 F:	include/imagemap.h
 F:	test/boot/imagemap.c
 
doc/develop/imagemap.rst new file +100 -0

100 lines not on this page · raw message

doc/develop/index.rst +1 -0
index 0b757e8601e..3b4cc2e28a7 100644
@@ -31,6 +31,7 @@ Implementation
 
    directories
    bloblist
+   imagemap
    bootstd/index
    ci_testing
    commands
Signature · 1 line
2.55.0
#12 lore raw
In reply to Daniel Golle
Quoted 9 lines · Daniel Golle

This series adds "imagemap", a small layer for reading FIT images directly from storage on demand instead of copying the whole image into RAM first. It keeps a translation table of the byte ranges already loaded, so a header probe, signature verification and the final load can reuse a single read, and it hooks transparently into fit_image_load() so all existing verification and decompression paths run unchanged for both RAM- and storage-backed images.

[...]

Applied to u-boot/next, thanks!

[01/11] mtd: bind an mtd_blk device for non-NAND MTD masters

        commit: e8a7453824b7ec20c722b0268de417058e49a9e0
[02/11] mtd: nand: sandbox: forbid sub-page writes
        commit: 7c821fea086014d3a78130a853e017a269d63e1e
[03/11] cmd: ubi: create a ubi_blk device when attaching UBI
        commit: dec405d1653a9e6b8bfc9a0f9aaf66ac3de7cb99
[04/11] mtd: nand: spi: drop the eager ubi_blk binding
        commit: 790c94a5c40a1b2c8cbcb87bc49c7fe0af5cbf2e
[05/11] mtd: ubi: block: build only with CONFIG_CMD_UBI
        commit: 15358294f11268e7998271df6e03127650178aa4
[06/11] mtd: ubi: block: only claim UBI block devices in the partition scan
        commit: d6bcfe912757ffc53403578700919a7a427ef94d
[07/11] boot: add imagemap on-demand loading from storage
        commit: 9b41c7dbee964fd6ea67c8ef23d30346c1f0f8a8
[08/11] boot: fit: support on-demand loading in fit_image_load()
        commit: 46d32e38ee4ef49567255aa9a89a7be44c0f0e04
[09/11] configs: sandbox: enable imagemap and its storage backends
        commit: feeef0a69f0f69252e8d06d906b80aaafe0c2cd2
[10/11] test: boot: add imagemap unit tests
        commit: 05c1fbbfa79ffe020b51b5d9ac9fd46ca3eb3e74
[11/11] doc: imagemap: document on-demand loading framework
        commit: 0f59e0702ae90dab8561e4ff2b32e89ef8390f28
Signature · 1 line
Tom