boot: add imagemap on-demand image loading from storage
Subsystems (2)
cover boot: add imagemap on-demand image loading from storage
This series adds "imagemap", a small layer for reading FIT images directly from storage on demand instead of copying the whole image into RAM first. It keeps a translation table of the byte ranges already loaded, so a header probe, signature verification and the final load can reuse a single read, and it hooks transparently into fit_image_load() so all existing verification and decompression paths run unchanged for both RAM- and storage-backed images.
An imagemap device is created over a partition of a block device. That covers more than raw disks: an MTD partition is reached through mtdblock and a UBI volume through ubiblock, each exposed as a named block-device partition, so imagemap carries no storage-specific code of its own. NAND bad-block handling and wear-levelling stay with UBI, as they do elsewhere in U-Boot; on NOR a partition is read linearly with end-to-end integrity guaranteed by the FIT hash or signature.
To make those two paths reachable and testable, the first patches fill in the missing block devices: a non-NAND MTD master now gets an mtd_blk device automatically (patch 1), and attaching UBI now binds a ubi_blk device on any flash (patch 3, which makes the SPI-NAND-only eager binding redundant, dropped in patch 4).
The read path is built on the SPL struct spl_load_info abstraction and a new spl_load_region() helper. imagemap is full-U-Boot only; the SPL FIT loader is unaffected.
Testing (sandbox): "ut imagemap" (13 tests) covers the translation-table core plus real end-to-end reads of a NOR partition through mtd_blk and a UBI volume through ubiblock; a CONFIG_IMAGEMAP=n build is unaffected.
On hardware it has been tested end-to-end on a BananaPi R3 (mt7986), booting a dm-verity-protected rootfs from every storage path the board
exposes: eMMC and SD (block partition), SPI-NAND (UBI volume via ubiblock) and SPI-NOR (raw MTD partition via mtdblock). The first in-tree consumer, a bootmeth for OpenWrt-style FIT firmware, will follow as a separate series.
This series began as part of a combined RFC that also carried the OpenWrt boot method and its bootdevs [1]. Following the review there it has been reworked substantially:
- Build on U-Boot's existing SPL external-data loading path rather than a new abstraction: imagemap now uses struct spl_load_info and a new spl_load_region() helper (suggested by Marek Vasut).
- Do not extend the bootm command with a storage source; on-demand loading hooks into fit_image_load() transparently, and the in-tree consumer is a bootstd boot method reached through bootflow/bootdev/bootmeth rather than new bootm syntax (per Tom Rini).
- Move the storage-backed path out of the long fit_image_load() into its own helper, gated with if()/tools_build() instead of an inline #if (per Simon Glass).
- Collapse the three storage-specific backends (block, MTD and UBI) into a single block backend: NOR is read through mtd_blk and SPI-NAND through UBI/ubiblock, so imagemap holds no storage-specific code.
- Send the on-demand loading core on its own; the OpenWrt boot method, the MTD/UBI bootdevs and board enablement follow as separate series once this lands.
In order to test imagemap end-to-end with the intended bootmeth user I'm still relying on the downstream install-uuid mechanism of [2], which [3] could supersede. Both have stalled right now.
[1] https://lore.kernel.org/u-boot/[email protected]/
[2] https://github.com/open-source-firmware/flat-image-tree/pull/40
[3] https://github.com/open-source-firmware/flat-image-tree/pull/44
Based on u-boot/next.
Daniel Golle (9):
mtd: bind an mtd_blk device for non-NAND MTD masters
mtd: nand: sandbox: forbid sub-page writes
cmd: ubi: create a ubi_blk device when attaching UBI
mtd: nand: spi: drop the eager ubi_blk binding
boot: add imagemap on-demand loading from storage
boot: fit: support on-demand loading in fit_image_load()
configs: sandbox: enable imagemap and its storage backends
test: boot: add imagemap unit tests
doc: imagemap: document on-demand loading framework
MAINTAINERS | 9 + arch/sandbox/dts/test.dts | 29 ++ boot/Kconfig | 16 + boot/Makefile | 2 + boot/bootm.c | 70 +++- boot/image-fit.c | 109 +++++- boot/imagemap.c | 451 +++++++++++++++++++++ cmd/ubi.c | 27 ++ configs/sandbox_defconfig | 6 + doc/develop/imagemap.rst | 100 +++++ doc/develop/index.rst | 1 + drivers/mtd/mtdcore.c | 24 ++ drivers/mtd/nand/raw/sand_nand.c | 2 + drivers/mtd/nand/spi/core.c | 3 - include/bootm.h | 2 + include/dm/uclass-id.h | 1 + include/image.h | 2 + include/imagemap.h | 136 +++++++ include/spl.h | 47 ++- test/boot/Makefile | 2 + test/boot/imagemap.c | 645 +++++++++++++++++++++++++++++++ test/cmd_ut.c | 2 + 22 files changed, 1671 insertions(+), 15 deletions(-) create mode 100644 boot/imagemap.c create mode 100644 doc/develop/imagemap.rst create mode 100644 include/imagemap.h create mode 100644 test/boot/imagemap.c
base-commit: 8841b7b292c74864acf4ed0bd8ce05839252c1f2
Signature · 1 line
2.55.0
PATCH 1/9 mtd: bind an mtd_blk device for non-NAND MTD masters
Only the SPI-NAND bind path currently creates an mtd_blk block device, so a bare NOR (parallel NOR, SPI-NOR, ...) has no block-device access at all. Bind an mtd_blk device for every non-NAND MTD master when CONFIG_MTD_BLOCK is enabled, so its partitions become reachable through the block layer (and thus by bootstd, imagemap, ...).
NAND is excluded on purpose: it needs UBI, which brings its own block device. Partitions inherit the master's block device via the "mtd" partition type, so only masters are bound.
mtd_bind() keeps a pointer to the passed mtd_info pointer (SPI-NAND fills it after bind), so the generic caller provides a small heap slot that lives as long as the block device; an MTD master is never removed in practice.
--- drivers/mtd/mtdcore.c | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+)
drivers/mtd/mtdcore.c +24 -0
@@ -494,6 +494,30 @@ int add_mtd_device(struct mtd_info *mtd)#endif mutex_unlock (&mtd_table_mutex); + +if (IS_ENABLED (CONFIG_MTD_BLOCK ) && mtd->dev && + !mtd_is_partition (mtd) && !mtd_type_is_nand (mtd)) { +/* +* Expose a non-NAND MTD master (parallel NOR, SPI-NOR, ...) as +* a mtd_blk block device, so its partitions are reachable +* through the block layer (and thus by imagemap, bootstd, ...). +* NAND is excluded: it needs UBI, which brings its own block +* device. Partitions inherit the master's block device via the +* "mtd" partition type, so only masters are bound. +* +* mtd_bind() keeps a pointer to the passed mtd_info pointer, so +* it needs storage that lives as long as the block device; the +* MTD master is never removed in practice, so a small heap slot +* is fine. +*/ +struct mtd_info **mtdp =kmalloc (sizeof (*mtdp),GFP_KERNEL ); + +if (mtdp) { + *mtdp = mtd; +mtd_bind (mtd->dev, mtdp); + } + } +/* We _know_ we aren't being removed, because our caller is still holding us here. So none of this try_ nonsense, and no bitching about it
Signature · 1 line
2.55.0
PATCH 2/9 mtd: nand: sandbox: forbid sub-page writes
In reply to Daniel GolleThe sandbox NAND model programs each page in a single operation and rejects a second write to an already-programmed page. UBI writes the EC and VID headers into the first page in two separate sub-page operations, which the model fails with -EIO, so UBI cannot attach to a sandbox NAND.
Advertise NAND_NO_SUBPAGE_WRITE so callers place each header in its own page, matching what the model can actually do. This lets the imagemap UBI-volume unit test attach UBI on the sandbox NAND.
--- drivers/mtd/nand/raw/sand_nand.c | 2 ++ 1 file changed, 2 insertions(+)
drivers/mtd/nand/raw/sand_nand.c +2 -0
@@ -602,6 +602,8 @@ static int sand_nand_probe(struct udevice *dev) nand = &chip->nand; nand->options =not_xpl () ?0 :NAND_SKIP_BBTSCAN ; +/* Each page is programmed in a single operation only */ + nand->options |=NAND_NO_SUBPAGE_WRITE ; nand->flash_node = np; nand->dev_ready = sand_nand_dev_ready; nand->cmdfunc = sand_nand_command;
Signature · 1 line
2.55.0
PATCH 3/9 cmd: ubi: create a ubi_blk device when attaching UBI
In reply to Daniel GolleA UBI volume can be reached as a partition on the ubi_blk block device, but that device is only bound from spinand_bind(), so it is missing on every other flash type (raw NAND, SPI-NOR, ...).
Bind a ubi_blk device when a UBI partition is attached, unless one has already been created, parented to the MTD device UBI sits on. UBI volumes then become reachable as block-device partitions regardless of the underlying flash. The imagemap block path relies on this to read a UBI volume.
--- cmd/ubi.c | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+)
cmd/ubi.c +27 -0
@@ -20,7 +20,10 @@#include <mtd.h> #include <nand.h> #include <onenand_uboot.h> +#include <dm/device.h> #include <dm/devres.h> +#include <dm/root.h> +#include <dm/uclass-internal.h> #include <linux/mtd/mtd.h> #include <linux/mtd/partitions.h> #include <linux/err.h>
@@ -670,6 +673,29 @@ int ubi_detach(void)return 0 ; } +#if CONFIG_IS_ENABLED (UBI_BLOCK ) +static void ubi_blk_bind_once (void ) +{ +struct udevice *dev; + +/* +* A single ubi_blk device serves all volumes of the attached UBI +* device, the volume being selected through the block descriptor's +* hwpart. Bind one when UBI is attached -- unless a previous attach +* already did -- parented to the MTD device UBI sits on. +*/ +for (uclass_find_first_device (UCLASS_BLK , &dev); dev; +uclass_find_next_device (&dev)) { +if (dev->driver ==DM_DRIVER_GET (ubi_blk)) +return ; + } + +ubi_bind (ubi && ubi->mtd && ubi->mtd->dev ? ubi->mtd->dev :dm_root ()); +} +#else +static inline void ubi_blk_bind_once (void ) { } +#endif +int ubi_part (const char *part_name,const char *vid_header_offset) {struct mtd_info *mtd;
@@ -698,6 +724,7 @@ int ubi_part(const char *part_name, const char *vid_header_offset) } ubi = ubi_devices[0 ]; +ubi_blk_bind_once ();return 0 ; }
Signature · 1 line
2.55.0
PATCH 4/9 mtd: nand: spi: drop the eager ubi_blk binding
In reply to Daniel Gollespinand_bind() binds a ubi_blk device at bind time. Now that "cmd:
ubi: create a ubi_blk device when attaching UBI" binds one generically when UBI is attached -- with an identical result, only lazily and for any flash type -- the SPI-NAND-specific eager call is redundant.
Drop it to avoid the duplicate code path (a ubi_blk is unusable before UBI is attached anyway). The mtd_blk binding is kept: bare mtd_blk access to SPI-NAND remains available for users who want it.
--- drivers/mtd/nand/spi/core.c | 3 --- 1 file changed, 3 deletions(-)
drivers/mtd/nand/spi/core.c +0 -3
@@ -1654,9 +1654,6 @@ static int spinand_bind(struct udevice *dev)if (ret)return ret; } - -if (CONFIG_IS_ENABLED (UBI_BLOCK )) -return ubi_bind (dev); }return 0 ;
Signature · 1 line
2.55.0
PATCH 5/9 boot: add imagemap on-demand loading from storage
In reply to Daniel GolleIntroduce imagemap, a small layer that reads image data from a storage device on demand and keeps a translation table of the byte ranges already loaded into RAM, instead of copying the whole image up front.
An imagemap device (UCLASS_IMAGEMAP) is created over a partition of a block device with imagemap_create(); imagemap_map() then returns a RAM pointer for any byte range, reading it in on first access and reusing it on later accesses -- so a header probe, verification and the final load of the same range share a single read. Block devices cover more than raw disks: an MTD partition is reached through mtdblock and a UBI volume through ubiblock, each exposed as a named block-device partition, so imagemap needs no storage-specific code of its own.
The read path is built on the SPL struct spl_load_info abstraction and a new spl_load_region() helper (added to spl.h): the block reader reports the device block length and spl_load_region() performs the native sector alignment. imagemap_map_to() keeps a payload byte-exact at a caller's load address, reading the aligned middle straight to the destination and bouncing only the partial head/tail block.
Regions that are not placed at a caller-supplied address are allocated through the LMB allocator and released on cleanup; the translation table doubles as the registry of those allocations. imagemap is gated to the full-U-Boot phase, so SPL and TPL carry no new cost.
--- MAINTAINERS | 7 + boot/Kconfig | 16 ++ boot/Makefile | 2 + boot/imagemap.c | 451 +++++++++++++++++++++++++++++++++++++++++ include/dm/uclass-id.h | 1 + include/imagemap.h | 136 +++++++++++++ include/spl.h | 47 ++++- 7 files changed, 657 insertions(+), 3 deletions(-) create mode 100644 boot/imagemap.c create mode 100644 include/imagemap.h
MAINTAINERS +7 -0
@@ -1186,6 +1186,13 @@ S: Maintained F: drivers/timer/goldfish_timer.c F: include/goldfish_timer.h +IMAGEMAP +M: Daniel Golle <[email protected]> +L: [email protected] +S: Maintained +F: boot/imagemap.c +F: include/imagemap.h + INTERCONNECT M: Neil Armstrong <[email protected]> S: Maintained
boot/Kconfig +16 -0
@@ -1205,6 +1205,22 @@ config SYS_BOOT_RAMDISK_HIGH endmenu # Boot images +config IMAGEMAP + bool "On-demand image loading from storage" + depends on DM && LMB && BLK && PARTITIONS + help + Read image data from a storage device on demand instead of + copying the whole image into RAM first. A translation table maps + already-loaded byte ranges to their RAM addresses to avoid + redundant reads, and scratch allocations are managed through the + LMB allocator. + + Images are read from a partition on a block device, identified by + number or name. MTD partitions (via mtdblock) and UBI volumes + (via ubiblock) are reached as block-device partitions too. Used + by bootm when a storage device is specified instead of a RAM + address. + config DISTRO_DEFAULTS bool "(deprecated) Script-based booting of Linux distributions" select CMDLINE
boot/Makefile +2 -0
@@ -73,6 +73,8 @@ obj-$(CONFIG_$(PHASE_)BOOTMETH_VBE_SIMPLE_OS) += vbe_simple_os.o obj-$(CONFIG_$(PHASE_)BOOTMETH_ANDROID) += bootmeth_android.o +obj-$(CONFIG_$(PHASE_)IMAGEMAP) += imagemap.o + obj-$(CONFIG_$(PHASE_)BOOTMETH_VBE_ABREC) += vbe_abrec.o vbe_common.o obj-$(CONFIG_$(PHASE_)BOOTMETH_VBE_ABREC_FW) += vbe_abrec_fw.o obj-$(CONFIG_$(PHASE_)BOOTMETH_VBE_ABREC_OS) += vbe_abrec_os.o
boot/imagemap.c new file +451 -0
451 lines not on this page · raw message
include/dm/uclass-id.h +1 -0
@@ -75,6 +75,7 @@ enum uclass_id {UCLASS_HASH ,/* Hash device */ UCLASS_HWSPINLOCK ,/* Hardware semaphores */ UCLASS_HOST ,/* Sandbox host device */ +UCLASS_IMAGEMAP ,/* On-demand image loading from storage */ UCLASS_I2C ,/* I2C bus */ UCLASS_I2C_EEPROM ,/* I2C EEPROM device */ UCLASS_I2C_GENERIC, /* Generic I2C device */
include/imagemap.h new file +136 -0
136 lines not on this page · raw message
include/spl.h +44 -3
74 lines not on this page · raw message
Signature · 1 line
2.55.0
PATCH 6/9 boot: fit: support on-demand loading in fit_image_load()
In reply to Daniel GolleIntegrate storage-backed image loading transparently into the existing FIT flow so that all verification, decompression, and copy-to-load- address code runs unchanged for both RAM-backed and storage-backed images.
Hook imagemap into fit_image_get_data(): when gd->imagemap is set and the sub-image uses external data, call imagemap_map() to bring the payload into RAM instead of returning the unmapped fit + offset pointer. Every existing caller, including fit_image_verify() called from fit_image_select(), transparently gets valid data without any changes to the authentication path.
For uncompressed sub-images with a known load address, pre-populate the translation table via imagemap_map_to() before verification runs. This loads data directly to the final RAM destination; when fit_image_get_data() is later called, the translation table hit returns the same pointer, achieving zero-copy. Compressed images or those without a load address fall through to the lazy imagemap_map() path.
Filesystem sub-images protected by dm-verity are returned early and stay on storage, as the kernel verifies them at block level.
--- boot/bootm.c | 70 +++++++++++++++++++++++++++--- boot/image-fit.c | 109 ++++++++++++++++++++++++++++++++++++++++++++++- include/bootm.h | 2 + include/image.h | 2 + 4 files changed, 174 insertions(+), 9 deletions(-)
boot/bootm.c +63 -7
@@ -14,6 +14,7 @@#include <env.h> #include <errno.h> #include <fdt_support.h> +#include <imagemap.h> #include <irq_func.h> #include <lmb.h> #include <log.h>
@@ -147,7 +148,22 @@ static int boot_get_kernel(const char *addr_fit, struct bootm_headers *images,/* check image type, for FIT images get FIT kernel node */ *os_data = *os_len =0 ; - buf =map_sysmem (img_addr,0 ); +if (IS_ENABLED (CONFIG_IMAGEMAP ) && images->imagemap) { +/* +* Storage path: read enough bytes to detect the image +* format. genimg_get_kernel_addr_fit() above still +* parsed any #config / :subimage suffix so the FIT +* selection variables are populated. +*/ + buf =imagemap_map (images->imagemap,0 ,64 ); +if (IS_ERR (buf)) { +puts ("Cannot read image header from storage\n" ); +return PTR_ERR (buf); + } + img_addr =map_to_sysmem (buf); + }else { + buf =map_sysmem (img_addr,0 ); + }switch (genimg_get_format (buf)) {#if CONFIG_IS_ENABLED (LEGACY_IMAGE_FORMAT )case IMAGE_FORMAT_LEGACY :
@@ -193,6 +209,20 @@ static int boot_get_kernel(const char *addr_fit, struct bootm_headers *images,#endif #if CONFIG_IS_ENABLED (FIT )case IMAGE_FORMAT_FIT : +if (IS_ENABLED (CONFIG_IMAGEMAP ) && images->imagemap) { +/* +* Extend the mapping to cover the full FIT +* FDT structure so all metadata is accessible. +*/ +size_t fdt_sz =fdt_totalsize (buf); + + buf =imagemap_map (images->imagemap,0 , fdt_sz); +if (IS_ERR (buf)) { +puts ("Cannot read FIT header from storage\n" ); +return PTR_ERR (buf); + } + img_addr =map_to_sysmem (buf); + } os_noffset = fit_image_load(images, img_addr, &fit_uname_kernel, &fit_uname_config, IH_ARCH_DEFAULT, IH_TYPE_KERNEL,
@@ -1033,11 +1063,21 @@ int bootm_run_states(struct bootm_info *bmi, int states)* Work through the states and see how far we get. We stop on * any error. */ -if (states &BOOTM_STATE_START ) +if (states &BOOTM_STATE_START ) { ret =bootm_start (); +/* +* bootm_start() zeroes the global images struct. Restore +* the loader pointer so the storage-backed path works. +*/ +if (IS_ENABLED (CONFIG_IMAGEMAP ) && bmi->imagemap) + images->imagemap = bmi->imagemap; + } -if (!ret && (states &BOOTM_STATE_PRE_LOAD )) - ret =bootm_pre_load (bmi->addr_img); +if (!ret && (states &BOOTM_STATE_PRE_LOAD )) { +/* Pre-load verification is not applicable to storage boot */ +if (!IS_ENABLED (CONFIG_IMAGEMAP ) || !images->imagemap) + ret =bootm_pre_load (bmi->addr_img); + }if (!ret && (states &BOOTM_STATE_FINDOS )) ret =bootm_find_os (bmi->cmd_name, bmi->addr_img);
@@ -1045,8 +1085,11 @@ int bootm_run_states(struct bootm_info *bmi, int states)if (!ret && (states &BOOTM_STATE_FINDOTHER )) {ulong img_addr; - img_addr = bmi->addr_img ?hextoul (bmi->addr_img,NULL ) - : image_load_addr; +if (IS_ENABLED (CONFIG_IMAGEMAP ) && images->imagemap) + img_addr = images->os.start; +else + img_addr = bmi->addr_img ?hextoul (bmi->addr_img,NULL ) + : image_load_addr; ret =bootm_find_other (img_addr, bmi->conf_ramdisk, bmi->conf_fdt); }
@@ -1145,11 +1188,24 @@ int bootm_run_states(struct bootm_info *bmi, int states) }/* Now run the OS! We hope this doesn't return */ -if (!ret && (states &BOOTM_STATE_OS_GO )) +if (!ret && (states &BOOTM_STATE_OS_GO )) { +/* Release storage backend before jumping — no return expected */ +if (IS_ENABLED (CONFIG_IMAGEMAP ) && images->imagemap) { +imagemap_cleanup (images->imagemap); + images->imagemap =NULL ; + } + ret =boot_selected_os (BOOTM_STATE_OS_GO , bmi, boot_fn); + }/* Deal with any fallout */ err: +/* Clean up imagemap on error (not reached on successful boot) */ +if (IS_ENABLED (CONFIG_IMAGEMAP ) && images->imagemap) { +imagemap_cleanup (images->imagemap); + images->imagemap =NULL ; + } +if (iflag)enable_interrupts ();
boot/image-fit.c +107 -2
139 lines not on this page · raw message
include/bootm.h +2 -0
@@ -40,6 +40,7 @@ struct cmd_tbl;* boot_get_fdt() for processing, or NULL for none * @boot_progress: true to show boot progress * @images: images information +* @imagemap: imagemap device for storage-backed boot (NULL for in-memory) * @cmd_name: command which invoked this operation, e.g. "bootm" * @argc: Number of arguments to the command (excluding the actual command). * This is 0 if there are no arguments
@@ -51,6 +52,7 @@ struct bootm_info {const char *conf_fdt;bool boot_progress;struct bootm_headers *images; +struct udevice *imagemap;const char *cmd_name;int argc;char *const *argv;
include/image.h +2 -0
@@ -412,6 +412,8 @@ struct bootm_headers {int verify;/* env_get("verify")[0] != 'n' */ +struct udevice *imagemap;/* on-demand storage loader, or NULL */ +#define BOOTM_STATE_START 0x00000001#define BOOTM_STATE_FINDOS 0x00000002#define BOOTM_STATE_FINDOTHER 0x00000004
Signature · 1 line
2.55.0
PATCH 7/9 configs: sandbox: enable imagemap and its storage backends
In reply to Daniel GolleEnable CONFIG_IMAGEMAP and, so the end-to-end unit tests can exercise the real block paths, the storage that feeds them: CONFIG_MTD_BLOCK and CONFIG_SPI_FLASH_MTD (a bare non-NAND MTD reached via mtd_blk) and CONFIG_CMD_UBI / CONFIG_UBI_BLOCK (a UBI volume reached via ubi_blk).
Give the sandbox SPI-NOR two device-tree partitions for the mtd_blk test and add a clean sandbox NAND chip (nand@2, no injected errors) so UBI can format on it for the ubiblock test.
--- arch/sandbox/dts/test.dts | 29 +++++++++++++++++++++++++++++ configs/sandbox_defconfig | 6 ++++++ 2 files changed, 35 insertions(+)
arch/sandbox/dts/test.dts +29 -0
@@ -1623,6 +1623,22 @@ compatible = "spansion,m25p16", "jedec,spi-nor"; spi-max-frequency = <40000000>; sandbox,filename = "spi.bin"; + + partitions { + compatible = "fixed-partitions"; + #address-cells = <1>; + #size-cells = <1>; + + partition@0 { + label = "nor-fit"; + reg = <0x0 0x100000>; + }; + + partition@100000 { + label = "nor-spare"; + reg = <0x100000 0x100000>; + }; + }; }; spi.bin@1 { reg = <1>;
@@ -2232,6 +2248,19 @@ sandbox,err-count = <3>; sandbox,err-step-size = <512>; }; + + /* Clean chip (no injected errors) for UBI/ubiblock tests */ + nand@2 { + reg = <2>; + nand-ecc-mode = "soft"; + sandbox,id = [00 e3]; + sandbox,erasesize = <(8 * 1024)>; + sandbox,oobsize = <16>; + sandbox,pagesize = <512>; + sandbox,pages = <0x2000>; + sandbox,err-count = <0>; + sandbox,err-step-size = <512>; + }; }; graph1 {
configs/sandbox_defconfig +6 -0
@@ -30,6 +30,7 @@ CONFIG_BOOTMETH_RAUC=y CONFIG_UPL=y CONFIG_LEGACY_IMAGE_FORMAT=y CONFIG_MEASURED_BOOT=y +CONFIG_IMAGEMAP=y CONFIG_BOOTSTAGE=y CONFIG_BOOTSTAGE_REPORT=y CONFIG_BOOTSTAGE_FDT=y
@@ -399,3 +400,8 @@ CONFIG_UTHREAD=y CONFIG_UNIT_TEST=y CONFIG_UT_TIME=y CONFIG_UT_DM=y +CONFIG_MTD_BLOCK=y +CONFIG_SPI_FLASH_MTD=y +CONFIG_CMD_UBI=y +CONFIG_MTD_UBI=y +CONFIG_UBI_BLOCK=y
Signature · 1 line
2.55.0
PATCH 8/9 test: boot: add imagemap unit tests
In reply to Daniel GolleAdd a test suite exercising imagemap:
- the translation-table core (map, lookup, map_to, cleanup, region growth, the block-aligned read path, and edge cases) using a mock RAM-backed reader, so most tests need no real storage;
- a NOR read through mtd_blk: write a known pattern into a device-tree partition of the sandbox SPI-NOR and read it back through imagemap, exercising imagemap -> spl_load_region -> blk_dread -> mtd_blk_read -> mtd_read for real;
- a UBI-volume read through ubiblock: attach UBI on the sandbox NAND, create a volume, write a pattern and read it back through imagemap on the ubi_blk device.
Registered under ut_boot_imagemap in cmd_ut.c.
--- MAINTAINERS | 1 + test/boot/Makefile | 2 + test/boot/imagemap.c | 645 +++++++++++++++++++++++++++++++++++++++++++ test/cmd_ut.c | 2 + 4 files changed, 650 insertions(+) create mode 100644 test/boot/imagemap.c
MAINTAINERS +1 -0
@@ -1192,6 +1192,7 @@ L: [email protected] S: Maintained F: boot/imagemap.c F: include/imagemap.h +F: test/boot/imagemap.c INTERCONNECT M: Neil Armstrong <[email protected]>
test/boot/Makefile +2 -0
@@ -27,3 +27,5 @@ endif obj-$(CONFIG_BOOTMETH_VBE) += vbe_fixup.o obj-$(CONFIG_UPL) += upl.o + +obj-$(CONFIG_IMAGEMAP) += imagemap.o
test/boot/imagemap.c new file +645 -0
645 lines not on this page · raw message
test/cmd_ut.c +2 -0
@@ -73,6 +73,7 @@ SUITE_DECL(optee);SUITE_DECL (pci_mps);SUITE_DECL (seama);SUITE_DECL (setexpr); +SUITE_DECL (imagemap);SUITE_DECL (upl);static struct suite suites[] = {
@@ -102,6 +103,7 @@ static struct suite suites[] = {SUITE (pci_mps,"PCI Express Maximum Payload Size" ),SUITE (seama,"seama command parameters loading and decoding" ),SUITE (setexpr,"setexpr command" ), +SUITE (imagemap,"imagemap on-demand storage loading" ), SUITE(upl, "Universal payload support"), };
Signature · 1 line
2.55.0
PATCH 9/9 doc: imagemap: document on-demand loading framework
In reply to Daniel GolleDescribe the imagemap framework architecture: the translation table that tracks mapped regions and its phase-determined caching, reading from a block-device partition (including MTD via mtdblock and UBI via ubiblock), the integration with bootm for on-demand FIT sub-image loading, and the SPL build.
--- MAINTAINERS | 1 + doc/develop/imagemap.rst | 100 +++++++++++++++++++++++++++++++++++++++ doc/develop/index.rst | 1 + 3 files changed, 102 insertions(+) create mode 100644 doc/develop/imagemap.rst
MAINTAINERS +1 -0
@@ -1191,6 +1191,7 @@ M: Daniel Golle <[email protected]> L: [email protected] S: Maintained F: boot/imagemap.c +F: doc/develop/imagemap.rst F: include/imagemap.h F: test/boot/imagemap.c
doc/develop/imagemap.rst new file +100 -0
100 lines not on this page · raw message
doc/develop/index.rst +1 -0
@@ -31,6 +31,7 @@ Implementation directories bloblist + imagemap bootstd/index ci_testing commands
Signature · 1 line
2.55.0
Re: [PATCH 5/9] boot: add imagemap on-demand loading from storage
In reply to Daniel GolleQuoted 27 lines · Daniel Golle
Introduce imagemap, a small layer that reads image data from a storage device on demand and keeps a translation table of the byte ranges already loaded into RAM, instead of copying the whole image up front.
An imagemap device (
UCLASS_IMAGEMAP) is created over a partition of a block device withimagemap_create();imagemap_map()then returns a RAM pointer for any byte range, reading it in on first access and reusing it on later accesses -- so a header probe, verification and the final load of the same range share a single read. Block devices cover more than raw disks: an MTD partition is reached through mtdblock and a UBI volume through ubiblock, each exposed as a named block-device partition, so imagemap needs no storage-specific code of its own.The read path is built on the SPL struct
spl_load_infoabstraction and a newspl_load_region()helper (added tospl.h): the block reader reports the device block length andspl_load_region()performs the native sector alignment.imagemap_map_to()keeps a payload byte-exact at a caller's load address, reading the aligned middle straight to the destination and bouncing only the partial head/tail block.Regions that are not placed at a caller-supplied address are allocated through the LMB allocator and released on cleanup; the translation table doubles as the registry of those allocations. imagemap is gated to the full-U-Boot phase, so SPL and TPL carry no new cost.
Signed-off-by Daniel Golle <[email protected]>---
[snip]
include/spl.h typedef ulong (*spl_load_reader)(struct spl_load_info *load, ulong sector, · line 356
Quoted 12 lines
include/spl.h +4
-0@@ -350,10 +350,21 @@ typedef ulong (*spl_load_reader)(struct spl_load_info *load, ulong sector,* @phase: Image phase to load * @no_fdt_update: true to update the FDT with any loadables that are loaded */ +/* +* struct spl_load_info carries a device block length for the SPL block +* loaders (CONFIG_SPL_LOAD_BLOCK) and, in full U-Boot, for the imagemap +* on-demand loader (CONFIG_IMAGEMAP);
This part should be made clear in updates to the existing comment block.
Quoted 2 lines
imagemap is gated out of the xpl
+ * phases so SPL and TPL carry no new cost.
This doesn't need to be noted I think because..
include/spl.h typedef ulong (*spl_load_reader)(struct spl_load_info *load, ulong sector, · line 367
Quoted 11 lines
+ */
include/spl.h +6
-1+#if IS_ENABLED (CONFIG_SPL_LOAD_BLOCK ) || \ + (IS_ENABLED (CONFIG_IMAGEMAP ) && !defined(CONFIG_XPL_BUILD )) +#define SPL_LOAD_INFO_HAS_BL_LEN +#endif +struct spl_load_info {spl_load_reader read;void *priv; -#if IS_ENABLED (CONFIG_SPL_LOAD_BLOCK ) +#ifdef SPL_LOAD_INFO_HAS_BL_LEN
Why not just:
#if IS_ENABLED(CONFIG_SPL_LOAD_BLOCK) || CONFIG_IS_ENABLED(IMAGEMAP)
? Then for when someone says "Ah, I want to do all of this right in SPL" there's less changes.
Signature · 1 line
Tom
Quoted 7 lines · Daniel Golle
This series adds "imagemap", a small layer for reading FIT images directly from storage on demand instead of copying the whole image into RAM first. It keeps a translation table of the byte ranges already loaded, so a header probe, signature verification and the final load can reuse a single read, and it hooks transparently into
fit_image_load()so all existing verification and decompression paths run unchanged for both RAM- and storage-backed images.
As I was looking at the patches, I saw at least one checkpatch.pl alignment place to fix, so please cover those in v2.
Conceptually, I'm OK with this. I need to throw the series at a world build before/after and see what the fallout is on not-enabled platforms. I'm assuming that once this gets where I can throw Coverity at it, it will complain about a few things too.
Before posting v2, please throw this though CI too, I'm pretty sure I saw a documentation style problem that would blow up there. Thanks!
Signature · 1 line
Tom
Re: [PATCH 1/9] mtd: bind an mtd_blk device for non-NAND MTD masters
In reply to Daniel GolleHi Daniel,
Quoted 24 lines · Daniel Golle
mtd: bind an
mtd_blkdevice for non-NAND MTD mastersOnly the SPI-NAND bind path currently creates an
mtd_blkblock device, so a bare NOR (parallel NOR, SPI-NOR, ...) has no block-device access at all. Bind anmtd_blkdevice for every non-NAND MTD master whenCONFIG_MTD_BLOCKis enabled, so its partitions become reachable through the block layer (and thus by bootstd, imagemap, ...).NAND is excluded on purpose: it needs UBI, which brings its own block device. Partitions inherit the master's block device via the "mtd" partition type, so only masters are bound.
mtd_bind()keeps a pointer to the passedmtd_infopointer (SPI-NAND fills it after bind), so the generic caller provides a small heap slot that lives as long as the block device; an MTD master is never removed in practice.Signed-off-by Daniel Golle <[email protected]>
drivers/mtd/mtdcore.c| 24 ++++++++++++++++++++++++1 file changed, 24 insertions(+)
drivers/mtd/mtdcore.c int add_mtd_device(struct mtd_info *mtd) · line 499
Quoted 6 lines
drivers/mtd/mtdcore.c +6
-0@@ -494,6 +494,30 @@ int add_mtd_device(struct mtd_info *mtd) +struct mtd_info **mtdp =kmalloc (sizeof (*mtdp),GFP_KERNEL ); + +if (mtdp) { + *mtdp = mtd; +mtd_bind (mtd->dev, mtdp); + }
Two issues. The return value from mtd_bind() is dropped, so on failure mtdp is leaked (mtd_bind() only pr_err()s and returns; it doesn't free what was passed in). And if kmalloc() returns NULL the block device is silently not created, with no hint why partitions are unreachable. Please log a warning in both paths and free mtdp on the mtd_bind() failure path.
drivers/mtd/mtdcore.c int add_mtd_device(struct mtd_info *mtd) · line 498
Quoted 7 lines
drivers/mtd/mtdcore.c +5
-0@@ -494,6 +494,30 @@ int add_mtd_device(struct mtd_info *mtd) +* mtd_bind() keeps a pointer to the passed mtd_info pointer, so +* it needs storage that lives as long as the block device; the +* MTD master is never removed in practice, so a small heap slot +* is fine. +*/
The pointer-to-pointer indirection only exists because SPI-NAND fills its mtd in after bind. Every other caller (including this one) already knows the mtd, so the heap slot is pure overhead and, as the comment concedes, leaks if the master is ever removed. Since you're touching this path anyway, how about a second helper (say mtd_bind_master(struct udevice *, struct mtd_info *)) that stashes the mtd pointer itself, and leave the double-indirection variant to the one caller that needs it. What do you think?
Regards,
Simon
Re: [PATCH 2/9] mtd: nand: sandbox: forbid sub-page writes
In reply to Daniel GolleHi Daniel,
Quoted 8 lines · Daniel Golle
mtd: nand: sandbox: forbid sub-page writes
The sandbox NAND model programs each page in a single operation and rejects a second write to an already-programmed page. UBI writes the EC and VID headers into the first page in two separate sub-page operations, which the model fails with
-EIO, so UBI cannot attach to a sandbox NAND.
Quoted 8 lines
Advertise
NAND_NO_SUBPAGE_WRITEso callers place each header in its own page, matching what the model can actually do. This lets the imagemap UBI-volume unit test attach UBI on the sandbox NAND.Signed-off-by Daniel Golle <[email protected]>
drivers/mtd/nand/raw/sand_nand.c| 2 ++1 file changed, 2 insertions(+)
BTW, longer term it would be nicer for the sandbox model to actually support sub-page writes so we can exercise the code path UBI uses on real hardware, but that's a separate improvement.
Regards,
Simon
Re: [PATCH 3/9] cmd: ubi: create a ubi_blk device when attaching UBI
In reply to Daniel GolleHi Daniel,
Quoted 19 lines · Daniel Golle
cmd: ubi: create a
ubi_blkdevice when attaching UBIA UBI volume can be reached as a partition on the
ubi_blkblock device, but that device is only bound fromspinand_bind(), so it is missing on every other flash type (raw NAND, SPI-NOR, ...).Bind a
ubi_blkdevice when a UBI partition is attached, unless one has already been created, parented to the MTD device UBI sits on. UBI volumes then become reachable as block-device partitions regardless of the underlying flash. The imagemap block path relies on this to read a UBI volume.Signed-off-by Daniel Golle <[email protected]>
cmd/ubi.c| 27 +++++++++++++++++++++++++++1 file changed, 27 insertions(+)
cmd/ubi.c int ubi_detach(void) · line 674
Quoted 2 lines
cmd/ubi.c +2
-0@@ -670,6 +673,29 @@ int ubi_detach(void) +ubi_bind (ubi && ubi->mtd && ubi->mtd->dev ? ubi->mtd->dev :dm_root ()); +}
The return value from ubi_bind() is silently dropped, so a failed bind (e.g. -ENOMEM) leaves ubi_part() reporting success while there is no ubi_blk at all. Please propagate it, or at least log a warning.
cmd/ubi.c int ubi_detach(void) · line 683
Quoted 13 lines
cmd/ubi.c +11
-0@@ -670,6 +673,29 @@ int ubi_detach(void) +/* +* A single ubi_blk device serves all volumes of the attached UBI +* device, the volume being selected through the block descriptor's +* hwpart. Bind one when UBI is attached -- unless a previous attach +* already did -- parented to the MTD device UBI sits on. +*/ +for (uclass_find_first_device (UCLASS_BLK , &dev); dev; +uclass_find_next_device (&dev)) { +if (dev->driver ==DM_DRIVER_GET (ubi_blk)) +return ; + }
ubi_detach() doesn't unbind the ubi_blk, so after 'ubi part A; ubi detach; ubi part B' the existing ubi_blk stays parented to A's MTD device even though UBI now sits on B. Reads still work because get_ubi_device() returns ubi_devices[0], but the DM tree is left with a stale parent (and A's MTD isn't safe to remove). Either unbind in ubi_detach(), or re-parent here when the target MTD has changed.
Also, walking every UCLASS_BLK to detect a single ubi_blk is awkward. Since ubi_blk is always parented to an MTD device, iterating the children of that parent e.g. with device_find_first_child_by_uclass() (or tracking the bound udevice in a static pointer) would be more direct and avoid poking dev->driver.
Regards,
Simon
Re: [PATCH 6/9] boot: fit: support on-demand loading in fit_image_load()
In reply to Daniel GolleHi Daniel,
Quoted 30 lines · Daniel Golle
boot: fit: support on-demand loading in
fit_image_load()Integrate storage-backed image loading transparently into the existing FIT flow so that all verification, decompression, and copy-to-load- address code runs unchanged for both RAM-backed and storage-backed images.
Hook imagemap into
fit_image_get_data(): whengd->imagemapis set and the sub-image uses external data, callimagemap_map()to bring the payload into RAM instead of returning the unmapped fit + offset pointer. Every existing caller, includingfit_image_verify()called fromfit_image_select(), transparently gets valid data without any changes to the authentication path.For uncompressed sub-images with a known load address, pre-populate the translation table via
imagemap_map_to()before verification runs. This loads data directly to the final RAM destination; whenfit_image_get_data()is later called, the translation table hit returns the same pointer, achieving zero-copy. Compressed images or those without a load address fall through to the lazyimagemap_map()[...]
boot/bootm.c| 70 +++++++++++++++++++++++++++++++----boot/image-fit.c | 109 ++++++++++++++++++++++++++++++++++++++++++++++++++++++- include/bootm.h | 2 + include/image.h | 2 + 4 files changed, 174 insertions(+), 9 deletions(-)
boot/bootm.c static int boot_get_kernel(const char *addr_fit, struct bootm_headers *images, · line 148
Quoted 1 line
boot/bootm.c +1
-0@@ -147,7 +148,22 @@ static int boot_get_kernel(const char *addr_fit, struct bootm_headers *images, + buf =imagemap_map (images->imagemap,0 ,64 );
Where does 64 come from? A short comment or named constant would help
- genimg_get_format() has to distinguish a legacy header, a FIT FDT magic and possibly an Android header, each with different minimum sizes, so an unexplained 64 looks like a coincidence.
boot/bootm.c int bootm_run_states(struct bootm_info *bmi, int states) · line 1071
Quoted 11 lines
boot/bootm.c +8
-0@@ -1033,11 +1063,21 @@ int bootm_run_states(struct bootm_info *bmi, int states) +if (states &BOOTM_STATE_START ) { ret =bootm_start (); +/* +* bootm_start() zeroes the global images struct. Restore +* the loader pointer so the storage-backed path works. +*/ +if (IS_ENABLED (CONFIG_IMAGEMAP ) && bmi->imagemap) + images->imagemap = bmi->imagemap; + }
Restoring a field immediately after bootm_start() zeroes it is fragile
- any future state that also calls bootm_start() (BOOTM_STATE_RESTART already runs bootm_restart(), similar in spirit) will silently drop the pointer again. Please move this into bootm_start() itself, or teach bootm_start() to preserve imagemap across the reset, so the invariant is expressed in one place.
boot/bootm.c int bootm_run_states(struct bootm_info *bmi, int states) · line 1193
Quoted 8 lines
boot/bootm.c +6
-0@@ -1145,11 +1188,24 @@ int bootm_run_states(struct bootm_info *bmi, int states) +err: +/* Clean up imagemap on error (not reached on successful boot) */ +if (IS_ENABLED (CONFIG_IMAGEMAP ) && images->imagemap) { +imagemap_cleanup (images->imagemap); + images->imagemap =NULL ; + }
Several paths in bootm_run_states() return directly rather than goto err - 'if (images->no_os) return -ENOPKG;', 'if (ret) return ret;' after the OS-func lookup, and the 'ERROR: booting os …' return 1 all bypass this cleanup, so imagemap leaks on those exits. Please convert those to goto err, or drive the cleanup from a single place that runs on every exit.
boot/image-fit.c int fit_image_get_data(const void *fit, int noffset, const void **data, · line 1150
Quoted 12 lines
boot/image-fit.c +10
-0@@ -1140,6 +1141,16 @@ int fit_image_get_data(const void *fit, int noffset, const void **data, +#if !defined(USE_HOSTCC ) &&CONFIG_IS_ENABLED (IMAGEMAP ) +if (images.imagemap) { +void *mapped; + + mapped =imagemap_lookup (images.imagemap, + offset, len); +if (mapped) + *data = mapped; + } +#endif
Reaching into the global images from a low-level helper couples fit_image_get_data() to bootm state - until now this function has been usable purely from (fit, noffset). Please pass the imagemap in explicitly (as a parameter or via a small wrapper), which also keeps 'ut fit' style tests free of hidden global dependencies.
boot/image-fit.c static const char *fit_get_image_type_property(int ph_type) · line 2156
Quoted 8 lines
boot/image-fit.c +6
-0@@ -2140,6 +2151,84 @@ static const char *fit_get_image_type_property(int ph_type) +if (CONFIG_IS_ENABLED (FIT_VERITY )) { +u8 img_type; + +if (!fit_image_get_type (fit, noffset, &img_type) && + img_type ==IH_TYPE_FILESYSTEM && +fdt_subnode_offset (fit, noffset,"dm-verity" ) >=0 ) {
Please use FIT_VERITY_NODENAME from image.h rather than a literal dm-verity, so there is one place to change the name.
boot/image-fit.c static const char *fit_get_image_type_property(int ph_type) · line 2159
Quoted 11 lines
boot/image-fit.c +9
-0@@ -2140,6 +2151,84 @@ static const char *fit_get_image_type_property(int ph_type) +if (!fit_image_get_data_position (fit, noffset, &data_off)) { + external = true; + }else if (!fit_image_get_data_offset (fit, noffset, &data_off)) { + external = true; + data_off +=ALIGN (fdt_totalsize (fit),4 ); + } + +if (!external ||fit_image_get_data_size (fit, noffset, &data_sz)) +return 0 ;
This re-implements the data-position / data-offset / totalsize-align dance that fit_image_get_data() already does a few hundred lines up, and uses a slightly different expression (ALIGN(fdt_totalsize(fit), 4) vs ((fdt_totalsize(fit) + 3) & ~3)). Please extract a shared helper - otherwise a future fix in one path (e.g. the INT_MAX bounds check that fit_image_get_data() has) will silently miss this one.
boot/image-fit.c static const char *fit_get_image_type_property(int ph_type) · line 2156
Quoted 8 lines
boot/image-fit.c +6
-0@@ -2140,6 +2151,84 @@ static const char *fit_get_image_type_property(int ph_type) +if (img_comp ==IH_COMP_NONE && load_op !=FIT_LOAD_IGNORED && + !fit_image_get_load (fit, noffset, &img_load)) { +void *dst =map_sysmem (img_load, data_sz); + + mapped =imagemap_map_to (images->imagemap, data_off, data_sz, + dst);
map_sysmem() should really be paired with unmap_sysmem() - a no-op on most arches but not on sandbox, which is where the tests run.
boot/image-fit.c int fit_image_load(struct bootm_headers *images, ulong addr, · line 2455
Quoted 7 lines
boot/image-fit.c +3
-2@@ -2349,8 +2453,9 @@ int fit_image_load(struct bootm_headers *images, ulong addr, -printf (" Loading %s from 0x%08lx to 0x%08lx\n" , - prop_name, data, load); +if (!CONFIG_IS_ENABLED (IMAGEMAP ) || data != load) +printf (" Loading %s from 0x%08lx to 0x%08lx\n" , + prop_name, data, load);
This is a build-time gate, not a runtime one - any board with CONFIG_IMAGEMAP=y silently loses the 'Loading … from … to …' message for RAM-backed FITs whenever data happens to equal load, a user-visible regression for boards that don't use imagemap at boot. Please key the suppression on 'images->imagemap && data == load' so the message only disappears on the genuine zero-copy path.
include/image.h struct bootm_headers { · line 412
Quoted 3 lines
include/image.h +1
-0@@ -412,6 +412,8 @@ struct bootm_headers { +struct udevice *imagemap;/* on-demand storage loader, or NULL */
It might be worth making this (and the equivalent field in struct bootm_info) conditional on CONFIG_IMAGEMAP, in line with the surrounding fields - otherwise every board pays the pointer with the feature off.
Regards,
Simon
Re: [PATCH 5/9] boot: add imagemap on-demand loading from storage
In reply to Daniel GolleHi Daniel,
Quoted 47 lines · Daniel Golle
boot: add imagemap on-demand loading from storage
Introduce imagemap, a small layer that reads image data from a storage device on demand and keeps a translation table of the byte ranges already loaded into RAM, instead of copying the whole image up front.
An imagemap device (
UCLASS_IMAGEMAP) is created over a partition of a block device withimagemap_create();imagemap_map()then returns a RAM pointer for any byte range, reading it in on first access and reusing it on later accesses -- so a header probe, verification and the final load of the same range share a single read. Block devices cover more than raw disks: an MTD partition is reached through mtdblock and a UBI volume through ubiblock, each exposed as a named block-device partition, so imagemap needs no storage-specific code of its own.The read path is built on the SPL struct
spl_load_infoabstraction and a newspl_load_region()helper (added tospl.h): the block reader reports the device block length andspl_load_region()performs the native sector alignment.imagemap_map_to()keeps a payload byte-exact at a caller's load address, reading the aligned middle straight to the [...]
MAINTAINERS| 7 +boot/Kconfig | 16 ++ boot/Makefile | 2 + boot/imagemap.c | 451 +++++++++++++++++++++++++++++++++++++++++++++++++ include/dm/uclass-id.h | 1 + include/imagemap.h | 136 +++++++++++++++ include/spl.h | 47 +++++- 7 files changed, 657 insertions(+), 3 deletions(-)boot/imagemap.c +14
-0@@ -0,0 +1,451 @@ +void *imagemap_map_to (struct udevice *dev,loff_t img_offset, +ulong size,void *dst) +{ +struct imagemap_priv *priv =dev_get_uclass_priv (dev); +void *p; +int ret; + +/* If already mapped to this exact destination, return it */ + p =imagemap_lookup (dev, img_offset, size); +if (p && p == dst) +return p; + + ret =imagemap_read_exact (&priv->info, img_offset, size, dst); +if (ret) {
boot/imagemap.c line 24
Quoted 10 lines
boot/imagemap.c +10
-0+log_err ("imagemap: read failed at offset 0x%llx (size 0x%lx): %d\n" , + (unsigned long long )img_offset, size, ret); +return ERR_PTR (ret); + } + +if (!imagemap_record (dev, img_offset, size, dst, false)) +return ERR_PTR (-ENOMEM ); + +return dst; +}
If an earlier imagemap_map() has already recorded a region at the same img_offset with lmb_reserved=true, this calls imagemap_record() with lmb_reserved=false, overwriting the entry in place - so it looks like the LMB reservation is silently orphaned and never freed by imagemap_cleanup(). Please free the old reservation before overwriting or refuse the overwrite.
boot/imagemap.c line 10
Quoted 12 lines
boot/imagemap.c +10
-0@@ -0,0 +1,451 @@ +if (bl_len ==1 ) +return spl_load_region (info, off, size, dst) <0 ? -EIO :0 ; + +/* Partial head block: bounce and copy the wanted tail of the block */ +if (overhead) { +ALLOC_CACHE_ALIGN_BUFFER (u8, blk, bl_len); +ulong chunk =min (left, bl_len - overhead); + +if (info->read (info, cur - overhead, bl_len, blk) < bl_len) +return -EIO ;
ALLOC_CACHE_ALIGN_BUFFER() with a runtime bl_len expands to a stack VLA. For MMC that's 512 bytes, but a UBI/ubiblock leb_size or MTD erase-block-sized reader can push this into the KB range. Probably this is fine, but do you know what the upper limit will be?
Quoted 5 lines
boot/imagemap.c +3
-0@@ -0,0 +1,451 @@ +alist_for_each (r, &priv->regions) { +if (r->img_offset == base_off && r->size < read_size) { + addr =map_to_sysmem (r->ram);
boot/imagemap.c line 13
Quoted 10 lines
boot/imagemap.c +10
-0+ +/* Free old LMB reservation if we own it */ +if (r->lmb_reserved) +lmb_free (addr, +ALIGN (r->size,ARCH_DMA_MINALIGN ), +LMB_NONE ); + +/* Try to re-reserve at the same address with new size */ +if (lmb_alloc_mem (LMB_MEM_ALLOC_ADDR ,0 , &addr, + alloc_size,LMB_NONE )) {
Two concerns with this extend path:
1. If r->lmb_reserved is false (the region was recorded by imagemap_map_to() at a caller-supplied dst), this happily lmb_alloc_mem()s over that memory and then re-reads on top of it. Whatever the caller placed there is silently clobbered. Please skip the extend path in that case, or refuse to extend a non-LMB region.
2. Even in the LMB-owned case, when the in-place re-reservation succeeds the whole range gets re-read although the first r->size bytes are already valid - the whole point of the translation table is to avoid redundant reads. Only the newly-covered tail needs reading.
boot/imagemap.c line 2
Quoted 4 lines
boot/imagemap.c +2
-0@@ -0,0 +1,451 @@ + ret =device_bind_driver (desc->bdev,"imagemap" , + name && *name ? name :"imagemap" , &imdev);
Just to check - is the intent that only one imagemap device is ever active per block device? Two calls with name == NULL (or two different partitions selected by index) will bind two children both called imagemap under the same parent, which works but makes the DM tree confusing. How about deriving a unique name (e.g. including the partition number) or documenting the single-instance assumption in imagemap_create()'s kernel-doc ?
boot/imagemap.c line 4
Quoted 6 lines
boot/imagemap.c +4
-0@@ -0,0 +1,451 @@ +void *imagemap_lookup (struct udevice *dev,loff_t img_offset,ulong size) +{ +struct imagemap_priv *priv =dev_get_uclass_priv (dev); +const struct imagemap_region *r;
The long block comment is useful but says the second guard prevents underflow on LP64 'where ulong and loff_t have the same rank'. loff_t is signed long long, 64-bit everywhere U-Boot runs; on ILP32 it doesn't share rank with ulong. Please tighten the wording - the guard is needed unconditionally because size is unsigned. Also you could wrap the comment to 80cols so more of it fits on the screen.
Regards,
Simon
Re: [PATCH 7/9] configs: sandbox: enable imagemap and its storage backends
In reply to Daniel GolleHi Daniel,
Quoted 22 lines · Daniel Golle
configs: sandbox: enable imagemap and its storage backends
Enable
CONFIG_IMAGEMAPand, so the end-to-end unit tests can exercise the real block paths, the storage that feeds them:CONFIG_MTD_BLOCKandCONFIG_SPI_FLASH_MTD(a bare non-NAND MTD reached viamtd_blk) andCONFIG_CMD_UBI/CONFIG_UBI_BLOCK(a UBI volume reached viaubi_blk).Give the sandbox SPI-NOR two device-tree partitions for the
mtd_blktest and add a clean sandbox NAND chip (nand@2, no injected errors) so UBI can format on it for the ubiblock test.Signed-off-by Daniel Golle <[email protected]>
arch/sandbox/dts/test.dts| 29 +++++++++++++++++++++++++++++configs/sandbox_defconfig | 6 ++++++ 2 files changed, 35 insertions(+)configs/sandbox_defconfig +0
-0
configs/sandbox_defconfig CONFIG_UTHREAD=y · line 408
Quoted 8 lines
configs/sandbox_defconfig +5
-0@@ -400,3 +401,8 @@ CONFIG_UTHREAD=y CONFIG_UNIT_TEST=y CONFIG_UT_TIME=y CONFIG_UT_DM=y +CONFIG_MTD_BLOCK=y +CONFIG_SPI_FLASH_MTD=y +CONFIG_CMD_UBI=y +CONFIG_MTD_UBI=y +CONFIG_UBI_BLOCK=y
Tom does this anyway from time to time, but you could run make savedefconfig to get these in the right order - also CONFIG_MTD_UBI is implied by CONFIG_UBI_BLOCK
Quoted 11 lines
arch/sandbox/dts/test.dts +6
-0@@ -1633,6 +1633,22 @@ compatible = "spansion,m25p16", "jedec,spi-nor"; spi-max-frequency = <40000000>; sandbox,filename = "spi.bin"; + + partitions { + compatible = "fixed-partitions"; + #address-cells = <1>; + #size-cells = <1>; +
arch/sandbox/dts/test.dts line 1651
Quoted 10 lines
arch/sandbox/dts/test.dts +10
-0+ partition@0 { + label = "nor-fit"; + reg = <0x0 0x100000>; + }; + + partition@100000 { + label = "nor-spare"; + reg = <0x100000 0x100000>; + }; + };
Adding fixed-partitions to spi.bin@0 combined with patch 1 (auto-bind mtd_blk on non-NAND masters) means every sandbox test now grows two extra block-device partitions on this flash. Please check the SPI bootdev tests (bootdev_test_next_prio in test/boot/bootdev.c expects only [email protected] / [email protected]) and dm_test_spi_flash in test/dm/sf.c still pass; if not, the partition table would be better hung off spi.bin@1 (which no existing test walks), or off a dedicated third flash chip.
Quoted 4 lines
arch/sandbox/dts/test.dts +2
-0@@ -2242,6 +2258,19 @@ + /* Clean chip (no injected errors) for UBI/ubiblock tests */ + nand@2 {
arch/sandbox/dts/test.dts line 2269
Quoted 10 lines
arch/sandbox/dts/test.dts +10
-0+ reg = <2>; + nand-ecc-mode = "soft"; + sandbox,id = [00 e3]; + sandbox,erasesize = <(8 * 1024)>; + sandbox,oobsize = <16>; + sandbox,pagesize = <512>; + sandbox,pages = <0x2000>; + sandbox,err-count = <0>; + sandbox,err-step-size = <512>; + };
Just to check - sandbox,err-count = <0> is the default, so this and sandbox,err-step-size can be omitted. The comment can go too; the property values already say it.
Regards,
Simon
Re: [PATCH 8/9] test: boot: add imagemap unit tests
In reply to Daniel GolleHi Daniel,
Quoted 29 lines · Daniel Golle
test: boot: add imagemap unit tests
Add a test suite exercising imagemap:
- the translation-table core (map, lookup,
map_to, cleanup, region growth, the block-aligned read path, and edge cases) using a mock RAM-backed reader, so most tests need no real storage;- a NOR read through
mtd_blk: write a known pattern into a device-tree partition of the sandbox SPI-NOR and read it back through imagemap, exercising imagemap ->spl_load_region->blk_dread->mtd_blk_read->mtd_readfor real;- a UBI-volume read through ubiblock: attach UBI on the sandbox NAND, create a volume, write a pattern and read it back through imagemap on the
ubi_blkdevice.Registered under
ut_boot_imagemapincmd_ut.c.Signed-off-by Daniel Golle <[email protected]>
MAINTAINERS| 1 +test/boot/Makefile | 2 + test/boot/imagemap.c | 645 +++++++++++++++++++++++++++++++++++++++++++++++++++ test/cmd_ut.c | 2 + 4 files changed, 650 insertions(+)
test/cmd_ut.c SUITE_DECL(optee); · line 77
Quoted 5 lines
test/cmd_ut.c +1
-0@@ -73,6 +73,7 @@ SUITE_DECL(optee);SUITE_DECL (pci_mps);SUITE_DECL (seama);SUITE_DECL (setexpr); +SUITE_DECL (imagemap);SUITE_DECL (upl);
Please keep the list alphabetical - imagemap belongs between image_fdt and lib, not between setexpr and upl. Same for the SUITE() entry below.
Quoted 4 lines
test/boot/imagemap.c +2
-0@@ -0,0 +1,645 @@ +/* Test: map() re-reads when extending a region to a larger size */ +static int imagemap_test_map_extend (struct unit_test_state *uts)
...
test/boot/imagemap.c line 277
Quoted 11 lines
test/boot/imagemap.c +11
-0+ { +void *p3 =imagemap_map (dev,256 ,128 ); +ulong p2_end =map_to_sysmem (p2) +256 ; + +ut_assert (!IS_ERR (p3)); +ut_asserteq (2 , priv->regions.count); +/* New region must not overlap the extended first region */ +ut_assert (map_to_sysmem (p3) >= p2_end || +map_to_sysmem (p3) +128 <=map_to_sysmem (p2)); +ut_asserteq_mem (image +256 , p3,128 ); + }
Please drop the anonymous block - hoist p3/p2_end to the top of the function, or split into its own test case. The nested scope is unusual in U-Boot tests.
Quoted 10 lines
test/boot/imagemap.c +8
-0@@ -0,0 +1,645 @@ +/* Test: read beyond image size returns error */ +static int imagemap_test_read_oob (struct unit_test_state *uts) +{ +struct udevice *dev; +u8 image[IMAGE_SIZE ]; +void *p; + +ut_assertok (create_mock_loader (image,IMAGE_SIZE , &dev));
test/boot/imagemap.c line 18
Quoted 10 lines
test/boot/imagemap.c +10
-0+ +/* Attempt to map beyond the end of the image */ + p =imagemap_map (dev,IMAGE_SIZE -32 ,64 ); +ut_assert (IS_ERR (p)); + +/* map_to should also fail */ +u8 dst[64 ]; + + p =imagemap_map_to (dev,IMAGE_SIZE -32 ,64 , dst); +ut_assert (IS_ERR (p));
Please declare dst at the top of the function - mid-block declarations are still avoided in U-Boot.
test/boot/imagemap.c line 1
Quoted 3 lines
test/boot/imagemap.c +1
-0@@ -0,0 +1,645 @@ +IMAGEMAP_TEST (imagemap_test_map_basic,0 );
All of these bind and probe devices under dm_root(), and the end-to-end MTD/UBI tests rely on real DM state. Please pass UTF_DM (and UTF_SCAN_FDT for the MTD test that walks device-tree partitions) so DM state is reset between tests - otherwise a test that aborts before imagemap_cleanup() leaks a bound imagemap-test device into the next test and the second run fails on the name clash in device_bind_driver().
Quoted 3 lines
test/boot/imagemap.c +1
-0@@ -0,0 +1,645 @@ +static int imagemap_test_ubiblock (struct unit_test_state *uts)
...
test/boot/imagemap.c line 614
Quoted 4 lines
test/boot/imagemap.c +4
-0+/* Attach UBI by name; this binds the ubi_blk block device */ +ut_assertok (ubi_part ("nand2" ,NULL )); +ut_assertok (ubi_create_vol ("vol0" ,0x10000 , true,UBI_VOL_NUM_AUTO , + false));
There is no matching detach, so a second run of the suite (or another UBI-touching test afterwards) sees UBI already attached to nand2 with vol0 present, and ubi_part()/ubi_create_vol() will fail. Please detach UBI at the end (and ideally guard against the pre-attached case at the start) so the test is re-runnable.
Regards,
Simon
Hi Daniel,
Quoted 2 lines · Daniel Golle
The first in-tree consumer, a bootmeth for OpenWrt-style FIT firmware, will follow as a separate series.
When that comes, will you be able to add a test for the bootmeth?
Quoted 1 line
boot: fit: support on-demand loading in
fit_image_load()
Related: patch 6 adds a struct udevice *imagemap field to both struct bootm_info and struct bootm_headers, but nothing in this series sets either outside the tests. The FIT hook is dead code in the built tree.
Quoted 2 lines
On hardware it has been tested end-to-end on a BananaPi R3 (mt7986), booting a dm-verity-protected rootfs from every storage path the board exposes
Sounds good.
Regards,
Simon
Re: [PATCH 9/9] doc: imagemap: document on-demand loading framework
In reply to Daniel GolleHi Daniel,
Quoted 7 lines · Daniel Golle
doc: imagemap: document on-demand loading framework
Describe the imagemap framework architecture: the translation table that tracks mapped regions and its phase-determined caching, reading from a block-device partition (including MTD via mtdblock and UBI via ubiblock), the integration with bootm for on-demand FIT sub-image loading, and the SPL build.
The doc doesn't actually describe the SPL build - imagemap is full-U-Boot only. Please either drop that clause from the commit message, or add a short paragraph in imagemap.rst noting the SPL gating (why xpl phases carry no cost).
doc/develop/imagemap.rst line 100
Quoted 11 lines
Signed-off-by Daniel Golle <[email protected]>
MAINTAINERS| 1 +doc/develop/imagemap.rst | 100 +++++++++++++++++++++++++++++++++++++++++++++++ doc/develop/index.rst | 1 + 3 files changed, 102 insertions(+)doc/develop/imagemap.rst new file +3
-0+The ``bootmeth_openwrt`` boot method is the primary consumer of this +framework, using it to boot OpenWrt-style FIT firmware images stored +directly on raw storage.
bootmeth_openwrt isn't in-tree yet - so better as 'will be the first in-tree consumer' so the doc doesn't point at a missing symbol.
doc/develop/index.rst Implementation · line 37
Quoted 9 lines
doc/develop/index.rst +1
-0@@ -31,6 +31,7 @@ Implementation directories bloblist + imagemap bootstd/index ci_testing commands
The surrounding entries are alphabetical (bloblist, bootstd, chid, ci_testing, commands, ...) so imagemap belongs later - around global_data/init, not between bloblist and bootstd/index.
doc/develop/imagemap.rst line 3
Quoted 5 lines
doc/develop/imagemap.rst +3
-0@@ -0,0 +1,100 @@ +``imagemap_create(dev, name, part)`` + Create an imagemap device over a partition of block device *dev*, + selected by *name* or by index *part*.
The real signature also takes a struct udevice **devp out parameter (see include/imagemap.h). Please add it here so the doc matches the API.
doc/develop/imagemap.rst line 11
Quoted 13 lines
doc/develop/imagemap.rst +11
-0@@ -0,0 +1,100 @@ +Boot Integration +---------------- + +The ``imagemap`` framework integrates with the boot pipeline through +``bootm_run_states()`` in ``boot/bootm.c``: + +1. The imagemap device pointer is stored in ``struct bootm_info`` + and propagated to the global ``images`` structure. + +2. ``boot_get_kernel()`` uses ``imagemap_map()`` to read the image + header and FIT metadata without loading the full image into RAM.
boot_get_kernel() is a static helper in boot/bootm.c - a doc reader won't easily find it. Please either name the entry point a caller actually reaches (bootm_run_states() / bootm_find_os()) or note that it's internal. Same for the imagemap_map_to() reference under fit_image_load(): it would help to say which caller in patch 6 uses it.
Regards,
Simon
Hi Simon,
thank you for the extensive review of the series. Meanwhile v2 had already been sent and Tom has already picked it into 'next'. Apart from one issue (see below) I've addressed all correctness issues you have pointed out in a follow up series which I have sent now.
Quoted 8 lines · Simon Glass
Hi Daniel,
Quoted 2 lines · Daniel Golle
The first in-tree consumer, a bootmeth for OpenWrt-style FIT firmware, will follow as a separate series.
When that comes, will you be able to add a test for the bootmeth?
Yes. I've got that planned but it's not done yet. Compared to imagemap and the dm-verity integration (which are both covered by unit tests) the bootmeth by itself also isn't that spectacular ;)
Quoted 5 lines
Quoted 1 line
boot: fit: support on-demand loading in
fit_image_load()Related: patch 6 adds a struct udevice *imagemap field to both struct
bootm_infoand structbootm_headers, but nothing in this series sets either outside the tests. The FIT hook is dead code in the built tree.
You are right, that is the gap the bootmeth series is going to close, and until it lands the integration is carried on the strength of out-of-tree testing, which is not a good place for it to sit. I would rather fix that by getting the consumer in than by adding a synthetic caller.
Cheers
Daniel
Hi Daniel,
Quoted 7 lines · Daniel Golle
Hi Simon,
thank you for the extensive review of the series. Meanwhile v2 had already been sent and Tom has already picked it into 'next'. Apart from one issue (see below) I've addressed all correctness issues you have pointed out in a follow up series which I have sent now.
Great!
Quoted 21 lines · Simon Glass
Quoted 8 lines
Hi Daniel,
Quoted 2 lines · Daniel Golle
The first in-tree consumer, a bootmeth for OpenWrt-style FIT firmware, will follow as a separate series.
When that comes, will you be able to add a test for the bootmeth?
Yes. I've got that planned but it's not done yet. Compared to imagemap and the dm-verity integration (which are both covered by unit tests) the bootmeth by itself also isn't that spectacular ;)
Quoted 5 lines
Quoted 1 line
boot: fit: support on-demand loading in
fit_image_load()Related: patch 6 adds a struct udevice *imagemap field to both struct
bootm_infoand structbootm_headers, but nothing in this series sets either outside the tests. The FIT hook is dead code in the built tree.
Quoted 5 lines
You are right, that is the gap the bootmeth series is going to close, and until it lands the integration is carried on the strength of out-of-tree testing, which is not a good place for it to sit. I would rather fix that by getting the consumer in than by adding a synthetic caller.
Sounds good.
Regards,
Simon